Skip to content

Commit 716622c

Browse files
loganjLarryclaude
committed
fix(relay): bind the resume age to the read a view holds; catch up from the report's rows
A window's resume age now comes from the head read whose rows it took, not any head read of the channel (a hover prefetch counted). Thread views get the same age rule. A report with nothing continuous takes the rows the window holds then as its catch-up target, instead of a resume keeping the drop's rows: a routine batch re-send after a resume reads only the head again. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Logan Johnson <[email protected]>
1 parent 01d6486 commit 716622c

6 files changed

Lines changed: 176 additions & 48 deletions

File tree

‎docs/relay-queries.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1044,13 +1044,17 @@ When access ends, the relay leaves the channel out of its batch with no frame,
10441044
and the channel's cursor stays. A grant that restores access may reach the
10451045
broker after the batch's EOSE, for example in the membership replay of the
10461046
same reconnect, so the batch can first report that channel continuous. The grant then re-sends the batch, whose EOSE reports it
1047-
again with nothing continuous. A resume keeps the rows the window held at the
1048-
drop until the next drop, so that catch-up pages back to them, past any rows
1049-
the stream delivered since.
1050-
1051-
A retained window that was verified when the stream dropped, with a head read
1052-
started at most 15 minutes before the drop, and a thread view that was settled
1053-
then, skip the head, page and thread reads for such a channel: the replay already delivered every channel event since the drop,
1047+
again with nothing continuous. A report with nothing continuous makes its
1048+
catch-up page back to the rows the window holds when the report arrives, unless
1049+
a drop kept older ones: while the relay left the channel out, the stream
1050+
delivered nothing for it, so those are the rows of the drop, and rows it
1051+
delivers after the report are newer than the gap. After a resume that was
1052+
continuous, a later re-send of the batch therefore reads only the head.
1053+
1054+
A retained window that was verified when the stream dropped, and a thread view
1055+
that was settled then, each holding rows from a read started at most 15 minutes
1056+
before the drop, skip the head, page and thread reads for such a channel (a
1057+
head read the window never took, such as a prefetch, does not count): the replay already delivered every channel event since the drop,
10541058
including ones backdated within the resume lag, and the live stream had
10551059
delivered everything before it. The window becomes verified at once, and a
10561060
membership notice read that failed or was cut before the drop resumes (it reads

‎src/features/relay/live-session.test.ts‎

Lines changed: 115 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1862,19 +1862,20 @@ describe("a reconnect whose replay was continuous", () => {
18621862
}
18631863
});
18641864

1865-
it("catches up from the rows held at the drop when a resumed channel is reported again with nothing continuous", async () => {
1865+
it("catches up from the rows held at the report when a resumed channel is reported again with nothing continuous", async () => {
18661866
const h = reconnectHarness();
18671867
try {
18681868
const seen = await verifiedThenDropped(h);
18691869
await resume(h, [], true);
18701870
expect(h.reads()).toHaveLength(0);
1871-
// A live event after the resume, then a grant the broker learned of
1872-
// after the replay: the replay may have left the channel out.
1871+
// A grant the broker learned of after the replay: the relay may have
1872+
// left the channel out, so the stream delivered nothing for it since.
1873+
h.live.established("c");
1874+
await flush();
1875+
// A live event after the re-sent batch is live, before the catch-up reads.
18731876
const fresh = message(h.alice, "c", "fresh", 60);
18741877
h.live.receive([fresh]);
18751878
await flush();
1876-
h.live.established("c");
1877-
await flush();
18781879
const hidden = message(h.alice, "c", "hidden", 30),
18791880
newer = message(h.alice, "c", "newer", 50);
18801881
// The head holds the live row; the gap lies below it, above `seen`.
@@ -1891,16 +1892,16 @@ describe("a reconnect whose replay was continuous", () => {
18911892
}
18921893
});
18931894

1894-
it("keeps the rows held at the drop when the stream drops again before a retracted resume catches up", async () => {
1895+
it("keeps the rows held at the report when the stream drops again before a retracted resume catches up", async () => {
18951896
const h = reconnectHarness();
18961897
try {
18971898
const seen = await verifiedThenDropped(h);
18981899
await resume(h, [], true);
1900+
h.live.established("c");
1901+
await flush();
18991902
const fresh = message(h.alice, "c", "fresh", 60);
19001903
h.live.receive([fresh]);
19011904
await flush();
1902-
h.live.established("c");
1903-
await flush();
19041905
// The retraction's head is still outstanding when the stream drops again.
19051906
h.read();
19061907
await h.drop();
@@ -1927,6 +1928,32 @@ describe("a reconnect whose replay was continuous", () => {
19271928
}
19281929
});
19291930

1931+
it("catches up a resumed window from its current rows when the batch is later re-sent", async () => {
1932+
// A routine refresh after a no-read resume: the window is contiguous
1933+
// through the rows the stream delivered since, so the head reaching the
1934+
// newest of them ends the catch-up.
1935+
const h = reconnectHarness();
1936+
try {
1937+
const seen = await verifiedThenDropped(h);
1938+
await resume(h, [], true);
1939+
const first = message(h.alice, "c", "live 1", 61),
1940+
second = message(h.alice, "c", "live 2", 62),
1941+
third = message(h.alice, "c", "live 3", 63);
1942+
const live = [first, second, third];
1943+
h.live.receive(live);
1944+
await flush();
1945+
h.live.established("c");
1946+
await flush();
1947+
h.read().respond(h.page("head", [third, second], second));
1948+
await flush();
1949+
expect(h.reads()).toHaveLength(0);
1950+
expect(h.rows()).toEqual([seen.id, ...live.map((event) => event.id)]);
1951+
expect(h.window().freshness).toBe("verified");
1952+
} finally {
1953+
h.owner.dispose();
1954+
}
1955+
});
1956+
19301957
it("catches up as before when the replay was cut", async () => {
19311958
const h = reconnectHarness();
19321959
try {
@@ -1974,6 +2001,37 @@ describe("a reconnect whose replay was continuous", () => {
19742001
}
19752002
});
19762003

2004+
it("catches up a window whose only recent head read was a prefetch it never took, though its replay was continuous", async () => {
2005+
// The prefetch's rows went to the cached head, not the open window, so
2006+
// they cannot show the window holds the frame the relay dropped.
2007+
let clock = 1_700_000_000_000;
2008+
const h = reconnectHarness({ now: () => clock });
2009+
try {
2010+
const seen = message(h.alice, "c", "seen", 10);
2011+
await h.reconnect(seen);
2012+
h.read().respond(h.page("head", [seen], seen));
2013+
await flush();
2014+
expect(h.window().freshness).toBe("verified");
2015+
const lost = message(h.alice, "c", "frame the relay dropped", 20);
2016+
clock += 15 * 60_000 + 1;
2017+
h.owner.session.channels.prepare?.("c");
2018+
await flush();
2019+
h.read().respond(h.page("head", [lost, seen], seen));
2020+
await flush();
2021+
expect(h.rows()).toEqual([seen.id]);
2022+
await h.drop();
2023+
h.queries.length = 0;
2024+
await resume(h, [], true);
2025+
expect(h.window().freshness).toBe("cached");
2026+
h.read().respond(h.page("head", [lost, seen], seen));
2027+
await flush();
2028+
expect(h.rows()).toEqual([seen.id, lost.id]);
2029+
expect(h.window().freshness).toBe("verified");
2030+
} finally {
2031+
h.owner.dispose();
2032+
}
2033+
});
2034+
19772035
it("a live batch refresh re-reads the head, so a later continuous reconnect needs no read", async () => {
19782036
let clock = 1_700_000_000_000;
19792037
const h = reconnectHarness({ now: () => clock });
@@ -2144,6 +2202,55 @@ describe("a reconnect whose replay was continuous", () => {
21442202
}
21452203
});
21462204

2205+
it("refreshes an open thread whose last read is older than the replay, though its replay was continuous", async () => {
2206+
// A reply the relay dropped from the live stream before the replay starts
2207+
// reaches the view only through a read.
2208+
let clock = 1_700_000_000_000;
2209+
const h = reconnectHarness({ now: () => clock });
2210+
try {
2211+
const root = message(h.alice, "c", "root", 10);
2212+
await verifiedThenDropped(h);
2213+
await resume(h, [], true);
2214+
const thread = h.owner.session.thread("c", root.id);
2215+
const answer = async () => {
2216+
for (let quiet = 0; quiet < 3; ) {
2217+
await flush();
2218+
if (!h.wire.pending.length) quiet++;
2219+
else quiet = 0;
2220+
for (const entry of h.wire.pending.splice(0))
2221+
entry.respond(
2222+
entry.filters[0]?.ids?.includes(root.id) ? [root] : [],
2223+
);
2224+
}
2225+
};
2226+
const threadReads = () =>
2227+
h.queries.filter((filters) =>
2228+
filters.some(
2229+
(f) => f.ids?.includes(root.id) || f["#e"]?.includes(root.id),
2230+
),
2231+
);
2232+
void thread.refresh();
2233+
await answer();
2234+
expect(thread.snapshot().status).toBe("ready");
2235+
clock += 15 * 60_000;
2236+
await h.drop();
2237+
h.queries.length = 0;
2238+
await resume(h, [], true);
2239+
await answer();
2240+
expect(threadReads()).toEqual([]);
2241+
clock += 1;
2242+
await h.drop();
2243+
h.queries.length = 0;
2244+
await resume(h, [], true);
2245+
await answer();
2246+
expect(threadReads().length).toBeGreaterThan(0);
2247+
expect(thread.snapshot().status).toBe("ready");
2248+
thread.dispose();
2249+
} finally {
2250+
h.owner.dispose();
2251+
}
2252+
});
2253+
21472254
it.each(["a refresh during the outage", "a refresh between two drops"])(
21482255
"refreshes a thread after %s, though its channel's replay was continuous",
21492256
async (change) => {

‎src/features/relay/live.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,15 @@ export const RELAY_TIMESTAMP_DRIFT_SECS = 900;
7070
* pod can trail its peers), the client clock's skew from the relay's, and the
7171
* relay's replica read budget for the unread repair, which uses the same margin. */
7272
export const LIVE_RESUME_LAG = RELAY_TIMESTAMP_DRIFT_SECS + 60;
73+
/** A continuous replay starts `LIVE_RESUME_LAG` before the drop, so it closes
74+
* the gap only back to then. A live frame the relay dropped earlier (a full
75+
* send buffer drops it without a notice) is not in it; only a read after that
76+
* frame brings it back. So a view resumes with no read only when the read its
77+
* rows came from started at most this long (milliseconds) before the drop; any
78+
* other view reads again. The batch refresh reads each open window's head and
79+
* open thread about this often. The minute under the lag absorbs read latency
80+
* and clock skew. */
81+
export const RESUME_READ_AGE = (LIVE_RESUME_LAG - 60) * 1000;
7382
/** Signature proofs a live subscription keeps across its reconnects. Each resume
7483
* replays `LIVE_RESUME_LAG` of events the subscription already verified; a
7584
* fresh signature check costs about 1 ms of main thread per event, a proof hit

‎src/features/relay/session.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -215,6 +215,8 @@ export function createRelaySession(
215215
} = {},
216216
) {
217217
let closed = false;
218+
// Thread views share the store's clock; thread() shadows `options`.
219+
const storeNow = options.now;
218220
const lifetime = new AbortController();
219221
let uploadLifetime = new AbortController();
220222
function cancelUploads() {
@@ -2802,6 +2804,7 @@ export function createRelaySession(
28022804
visible: (events) => events.filter(visibility(events)),
28032805
notify,
28042806
dropped: streamDropped,
2807+
now: storeNow,
28052808
});
28062809
threads.add(thread);
28072810
thread.receive(recent.entries().map(([, item]) => item.event));

‎src/features/relay/store.ts‎

Lines changed: 21 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ import { yieldToHost } from "./yield";
4949
import { locate, SEEK_PAGE, type SeekRead } from "./history-seek";
5050
import type { WindowPage } from "./window";
5151
import { MatchRank, matchRank } from "../search/match";
52+
import { RESUME_READ_AGE } from "./live";
5253

5354
type Listener = () => void;
5455
type WindowState = {
@@ -87,15 +88,13 @@ type WindowState = {
8788
* gap, not overlap with it. */
8889
kept?: Set<string> | undefined;
8990
/** The window was verified when the stream dropped and `kept` was taken,
90-
* and a head read started at most `RESUME_READ_AGE` before: its rows reached
91+
* and its `readAt` was at most `RESUME_READ_AGE` before: its rows reached
9192
* the live stream then. A continuous replay of the channel closes the gap
9293
* from them, so `resume` needs no read. */
9394
keptLive?: boolean | undefined;
94-
/** `resume` verified the window with no read. `kept` still holds the rows
95-
* of the drop, so a later report on this connection that the channel was
96-
* not continuous (a grant the broker learned of after its replay) catches up
97-
* from them; the next drop takes a new snapshot. */
98-
resumed?: boolean | undefined;
95+
/** When the head read whose rows the window holds started. A head read the
96+
* window never took (a prefetch, a cancelled catch-up) does not count. */
97+
readAt?: number | undefined;
9998
/** Relay membership notices, read apart from the pages (see `fillNotices`). */
10099
notices: Map<string, RelayEvent>;
101100
/** Where the next older notice read starts: null once a read reached the
@@ -111,6 +110,8 @@ type Head = {
111110
hasMore: boolean;
112111
events: readonly RelayEvent[];
113112
savedAt: number;
113+
/** When the relay read that produced it started; none for a saved head. */
114+
readAt?: number;
114115
cached: boolean;
115116
restored?: true;
116117
};
@@ -149,15 +150,6 @@ const sameKeys = (a?: readonly string[], b?: readonly string[]) =>
149150
(a ?? []).every((key, index) => key === b?.[index]));
150151
/** Relay page size, separate from discovery's retained-entry budget. */
151152
const DISCOVERY_LIMIT = 500;
152-
/** A continuous replay starts `LIVE_RESUME_LAG` (16 minutes) before the drop,
153-
* so it closes the gap only back to then. A live frame the relay dropped
154-
* earlier (a full send buffer drops it without a notice) is not in it; only a
155-
* head read after that frame brings it back. The live batch refresh reads each
156-
* caught-up head about every 15 minutes. So a window resumes with no read only
157-
* when a head read started at most this long before the drop; any other
158-
* window catches up. The minute under the lag absorbs read latency and clock
159-
* skew. */
160-
const RESUME_READ_AGE = 15 * 60_000;
161153
/** Channels one creator-proof filter names. Each has one creation, so a limit
162154
* of twice this, the relay's maximum, leaves room for strays; a filter that
163155
* reaches it proves only the channels whose creation arrived. */
@@ -317,8 +309,6 @@ export function createChannelStore(
317309
// Full cache clears. Profiles outlive access purges (which move `epoch`),
318310
// so only a clear or disposal fences a profile restore.
319311
let cleared = 0;
320-
/** When the latest head read of each channel started (see `RESUME_READ_AGE`). */
321-
const headReads = new Map<string, number>();
322312
let listAgain = false;
323313
// Epoch of the latest access revocation. When that revocation is what made a
324314
// full read stale, it interrupted rather than failed it: a pass is still owed.
@@ -744,6 +734,7 @@ export function createChannelStore(
744734
moveCursor(state, head.cursor);
745735
state.atHead = true;
746736
state.restored = head.restored === true;
737+
state.readAt = head.readAt;
747738
if (dropped) {
748739
state.kept = rowIds(state);
749740
state.keptLive = false;
@@ -932,11 +923,9 @@ export function createChannelStore(
932923
});
933924
// Head reads belong to session recovery; only manual pagination needs a
934925
// finite retry owner here.
935-
const startedAt = now();
936926
const events = cursor
937927
? await retryRead(read, settings.signal ?? new AbortController().signal)
938928
: await read();
939-
if (!cursor) headReads.set(channelId, startedAt);
940929
return {
941930
events,
942931
page: parseWindow(channelId, cursor, transport.relayAuthor, events),
@@ -976,6 +965,7 @@ export function createChannelStore(
976965
hasMore: page.hasMore,
977966
events,
978967
savedAt: now(),
968+
readAt: startedAt,
979969
cached: false,
980970
};
981971
if (
@@ -4162,13 +4152,11 @@ export function createChannelStore(
41624152
for (const [id, head] of heads.entries())
41634153
heads.set(id, { ...head, cached: true });
41644154
for (const state of windows.values()) {
4165-
if (!state.kept || state.resumed) {
4166-
state.resumed = undefined;
4155+
if (!state.kept) {
41674156
state.kept = rowIds(state);
41684157
state.keptLive =
41694158
state.snapshot.freshness === "verified" &&
4170-
now() - (headReads.get(state.channelId) ?? -Infinity) <=
4171-
RESUME_READ_AGE;
4159+
now() - (state.readAt ?? -Infinity) <= RESUME_READ_AGE;
41724160
}
41734161
setWindow(state, {
41744162
freshness: "cached",
@@ -4225,7 +4213,6 @@ export function createChannelStore(
42254213
);
42264214
for (const id of kept) retained.traffic.delete(id);
42274215
retained.kept = keptAfter();
4228-
retained.resumed = undefined;
42294216
retained.floor = undefined;
42304217
retained.atHead = true;
42314218
moveCursor(retained, last.cursor);
@@ -4243,6 +4230,7 @@ export function createChannelStore(
42434230
if (!valid())
42444231
throw new DOMException("Stale live catch-up", "AbortError");
42454232
retained.restored = false;
4233+
retained.readAt = head.readAt;
42464234
accept(head.events);
42474235
page = head;
42484236
const reached = (last: Page) =>
@@ -4271,7 +4259,6 @@ export function createChannelStore(
42714259
throw new DOMException("Stale live catch-up", "AbortError");
42724260
if (retained.atHead) moveCursor(retained, page.cursor);
42734261
retained.kept = keptAfter();
4274-
retained.resumed = undefined;
42754262
// Live replay covers minutes; notices from a longer drop come by read.
42764263
void fillNotices(retained, true);
42774264
// A drop mid-fill has marked the window cached; the stream is retrying.
@@ -4328,8 +4315,8 @@ export function createChannelStore(
43284315
dropped = false;
43294316
const head = heads.peek(channelId);
43304317
if (head) heads.set(channelId, { ...head, cached: true });
4318+
state.kept = undefined;
43314319
state.keptLive = undefined;
4332-
state.resumed = true;
43334320
setWindow(state, { freshness: "verified", error: undefined });
43344321
// The replay holds only the outage's notices: finish a notice read that
43354322
// failed or was cut before the drop. It reads nothing when none is owed.
@@ -4366,11 +4353,15 @@ export function createChannelStore(
43664353
if (head) heads.set(channelId, { ...head, cached: true });
43674354
const state = windows.get(channelId);
43684355
if (state) {
4369-
// This establishment's replay may stop short of the kept rows. A
4370-
// resumed window owes a catch-up again: a drop before it settles keeps
4371-
// the rows of the first drop.
4356+
// This establishment's replay may stop short of the kept rows. A window
4357+
// with none (resumed with no read, or not dropped) takes the rows it
4358+
// holds now: they reached the stream, and rows it delivers from here on
4359+
// are newer than any gap. While the relay leaves a channel out (access
4360+
// ended, and the grant reached the broker after its replay), the stream
4361+
// delivers none for it, so these are still the rows of the drop. A drop
4362+
// before the catch-up settles keeps them.
4363+
state.kept ??= rowIds(state);
43724364
state.keptLive = false;
4373-
state.resumed = undefined;
43744365
setWindow(state, {
43754366
freshness: "cached",
43764367
error: undefined,

0 commit comments

Comments
 (0)