Skip to content

Commit 1833d19

Browse files
baxenmbot
andcommitted
fix(unread): raise a local read floor over dropped marks, and halve save validation
Read marks that the local reserve has no room for were deleted, which made what they read unread again: old mentions kept coming back until the 30-day floor passed them. The reserve now raises a local-only floor to the time of the newest mark it drops, keeps the newest marks so that floor rises as little as possible, and drops marks at or below it. Unread evaluation uses the later of the floor and the retention window; publication keeps the retention window only. Old builds ignore and drop the floor. Each save validated the stored record and then its own output. A handle now saves a random token with its record and reuses its own validated journal when it finds that token again; any other writer leaves it out. contextId skips UTF-8 encoding for strings too short to exceed its byte limit. Co-authored-by: mbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz> Signed-off-by: Bradley Axen <[email protected]>
1 parent ffd546e commit 1833d19

7 files changed

Lines changed: 178 additions & 30 deletions

File tree

‎docs/unread.md‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -473,10 +473,15 @@ A local-only `reserve` field in the same journal record keeps receipts evicted
473473
from the sync journal, up to 5,000 keys / 512 KiB of serialized JSON. It preserves
474474
actual frontiers, never manufactures channel cutoffs, and commits atomically with
475475
the journal. Local unread decisions use both sets. Returning keys take the maximum
476-
frontier before leaving the reserve. Its finite eviction order favors channel,
477-
thread, then catch-up receipts before individual messages; newest event timestamps
478-
win within each group. While overrides exist, inherited reserve floors stay protected
479-
and direct override floors return to the journal. Without overrides, the reserve
476+
frontier before leaving the reserve. It keeps the newest receipts by frontier time.
477+
A receipt it has no room for raises a local-only `floor` in the same record to that
478+
receipt's time: on this installation every message at or before the floor is read,
479+
as with the 30-day retention floor, so dropping a receipt never makes what it read
480+
unread again. Keeping the newest receipts raises the floor as little as possible.
481+
Receipts at or below the floor read nothing more and are dropped. Manual unread and
482+
overrides still win over the floor. Publication keeps only the retention floor. While
483+
overrides exist, inherited reserve floors stay protected, nothing is dropped for
484+
being under the floor, and direct override floors return to the journal. Without overrides, the reserve
480485
also drops a receipt that a kept broader receipt already covers (the same rule the
481486
journal uses), whether the cover is in the journal or the reserve.
482487

@@ -495,11 +500,12 @@ only the message says whether catch-up reads it. Old builds discard `homes` on
495500
their next save, as with the reserve.
496501

497502
This extends retention only on the same browser profile/install. It cannot recover
498-
already discarded receipts, prevent loss after exhausting the reserve, or improve a
499-
fresh profile's smaller synced copy. An automatic observation already covered by
503+
already discarded receipts or improve a fresh profile's smaller synced copy. Once the
504+
reserve is full, the floor trades old unread for capacity: a mention older than the
505+
floor that was never read counts as read on this installation. An automatic observation already covered by
500506
the reserve does not republish that receipt. Manual unread still wins. Old builds
501-
can load the unchanged sync journal but discard the optional reserve on their next
502-
save; community leave on any build deletes both together. No database migration
507+
can load the unchanged sync journal but discard the optional reserve and floor on
508+
their next save; community leave on any build deletes both together. No database migration
503509
or wire-format change is involved.
504510

505511
Override groups, permanent clear floors, directly associated frontiers and possible

‎src/features/relay/read-state-model.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,10 @@ export const uint32 = (value: unknown): value is number =>
2020
export const record = (value: unknown): value is Record<string, unknown> =>
2121
value !== null && typeof value === "object" && !Array.isArray(value);
2222
export const contextId = (value: unknown): value is string =>
23-
typeof value === "string" && value.length > 0 && bytes(value) <= 256;
23+
typeof value === "string" &&
24+
value.length > 0 &&
25+
// UTF-8 takes at most 3 bytes per UTF-16 unit: short strings need no encoding.
26+
(value.length <= 85 || bytes(value) <= 256);
2427
export const slotId = (value: unknown): value is string =>
2528
typeof value === "string" && /^[0-9a-f]{32}$/.test(value);
2629

‎src/features/relay/read-state-retention.test.ts‎

Lines changed: 48 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ it("a cover that replaces a fresh read keeps its recency at the synced limit", (
257257
expect(stillRead(published, marks)).toBe(true);
258258
});
259259

260-
it("bounds the local reserve by entries and bytes, dropping oldest receipts deterministically", () => {
260+
it("bounds the local reserve by entries and bytes, dropping oldest receipts deterministically and raising the floor over them", () => {
261261
for (const width of [64, 240]) {
262262
const frontiers = Object.fromEntries(
263263
Array.from({ length: 8000 }, (_, n) => [
@@ -278,8 +278,23 @@ it("bounds the local reserve by entries and bytes, dropping oldest receipts dete
278278
expect(frontiers[key]).toBe(value);
279279
expect(kept.state.frontiers[key]).toBeUndefined();
280280
}
281+
// Every dropped receipt is at or below the floor, so it stays read.
282+
const local = { ...kept.reserve, ...kept.state.frontiers };
283+
const dropped = Object.entries(frontiers).filter(
284+
([key]) => !(key in local),
285+
);
286+
expect(dropped.length).toBeGreaterThan(0);
287+
expect(kept.floor).toBe(Math.max(...dropped.map(([, value]) => value)));
288+
expect(Math.min(...Object.values(kept.reserve))).toBe(kept.floor + 1);
281289
expect(
282-
retainLocalRead([kept.state], kept.recent, "fixture", kept.reserve),
290+
retainLocalRead(
291+
[kept.state],
292+
kept.recent,
293+
"fixture",
294+
kept.reserve,
295+
undefined,
296+
kept.floor,
297+
),
283298
).toEqual(kept);
284299
}
285300
});
@@ -332,7 +347,7 @@ it("promotes archived override floors before reserve pressure can reactivate the
332347
).toBe(false);
333348
});
334349

335-
it("keeps archived quiet-channel catch-up ahead of newer message churn", () => {
350+
it("drops archived quiet-channel catch-up before newer message churn, under the floor", () => {
336351
const frontiers = Object.fromEntries(
337352
Array.from({ length: 8000 }, (_, n) => [
338353
`msg:${n.toString(16).padStart(64, "0")}`,
@@ -342,10 +357,39 @@ it("keeps archived quiet-channel catch-up ahead of newer message churn", () => {
342357
const kept = retainLocalRead([{ frontiers, overrides: {} }], {}, "fixture", {
343358
"activity:quiet": 1,
344359
});
345-
expect(kept.reserve["activity:quiet"]).toBe(1);
360+
expect(kept.reserve["activity:quiet"]).toBeUndefined();
346361
expect(kept.state.frontiers["activity:quiet"]).toBeUndefined();
362+
expect(kept.floor).toBeGreaterThanOrEqual(100);
347363
expect(Object.keys(kept.reserve)).toHaveLength(READ_RESERVE_KEYS);
348364
});
365+
it("drops marks at or below the floor, unless overrides make them load-bearing", () => {
366+
const state = { frontiers: { room: 10, "msg:new": 30 }, overrides: {} };
367+
const kept = retainLocalRead(
368+
[state],
369+
{},
370+
"fixture",
371+
{ "msg:old": 20 },
372+
undefined,
373+
20,
374+
);
375+
expect(kept.state.frontiers).toEqual({ "msg:new": 30 });
376+
expect(kept.reserve).toEqual({});
377+
expect(kept.floor).toBe(20);
378+
const overrides = { "msg:child": { set: 1, clear: 0, baseline: 0 } };
379+
const guarded = retainLocalRead(
380+
[{ ...state, overrides }],
381+
{},
382+
"fixture",
383+
{ "msg:old": 20 },
384+
undefined,
385+
20,
386+
);
387+
expect({ ...guarded.reserve, ...guarded.state.frontiers }).toEqual({
388+
room: 10,
389+
"msg:new": 30,
390+
"msg:old": 20,
391+
});
392+
});
349393
it("a large inherited reserve does not overflow the journal on remote override ingest", () => {
350394
const reserve = Object.fromEntries(
351395
Array.from({ length: 2000 }, (_, n) => [

‎src/features/relay/read-state-retention.ts‎

Lines changed: 44 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -241,32 +241,51 @@ export function retainReadState(
241241
/** Local-only receipts evicted from the sync journal. Never passed to publication. */
242242
export const READ_RESERVE_KEYS = 5000;
243243
export const READ_RESERVE_BYTES = 512 * 1024;
244+
/**
245+
* The local read set: the journal (`retainRead`) plus the reserve, with a
246+
* read-through `floor`. Every message at or before the floor counts as read
247+
* on this installation, so a mark that capacity drops raises the floor to its
248+
* own time rather than making what it read unread again. The reserve keeps
249+
* the newest marks, so the floor rises as little as it can. A mark at or
250+
* below the floor reads nothing more and is dropped, unless overrides make
251+
* inherited marks load-bearing.
252+
*/
244253
export function retainLocalRead(
245254
states: readonly ReadState[],
246255
recent: Readonly<Record<string, number>>,
247256
clientId: string,
248257
reserve: Readonly<Record<string, number>> = {},
249258
covered?: CoveredFrontier,
259+
floor = 0,
250260
) {
251-
const frontiers = new Map(Object.entries(reserve));
252261
const overrides = new Set(
253262
states.flatMap((state) => Object.keys(state.overrides)),
254263
);
264+
const below = (value: number) => !overrides.size && value <= floor;
265+
const frontiers = new Map(
266+
Object.entries(reserve).filter(([, value]) => !below(value)),
267+
);
255268
const returning: Record<string, number> = {};
256-
for (const state of states)
269+
const current: ReadState[] = [];
270+
for (const state of states) {
271+
const kept: Record<string, number> = {};
257272
for (const [key, value] of Object.entries(state.frontiers)) {
273+
if (below(value)) continue;
274+
kept[key] = value;
258275
const previous = frontiers.get(key);
259276
if (previous !== undefined) returning[key] = previous;
260277
frontiers.set(key, Math.max(previous ?? 0, value));
261278
}
279+
current.push({ frontiers: kept, overrides: state.overrides });
280+
}
262281
// Direct override floors belong in the journal. Possible inherited floors
263282
// stay protected in the reserve instead of overflowing the smaller journal.
264283
for (const key of overrides) {
265284
const value = frontiers.get(key);
266285
if (value !== undefined) returning[key] = value;
267286
}
268287
const kept = retainRead(
269-
[...states, { frontiers: returning, overrides: {} }],
288+
[...current, { frontiers: returning, overrides: {} }],
270289
recent,
271290
clientId,
272291
undefined,
@@ -278,8 +297,9 @@ export function retainLocalRead(
278297
const entries: [string, number][] = [];
279298
const accepted = new Map<string, number>();
280299
// A mark that a kept broader mark already reads is redundant here too, as
281-
// in the journal: keep it and it only takes room from real reads. Covers
282-
// always have a broader scope, so they are accepted before what they cover.
300+
// in the journal: keep it and it only takes room from real reads. A cover
301+
// reads through at least the time of what it covers, so newest first
302+
// accepts it before what it covers.
283303
// Overrides make inherited ancestry load-bearing; keep everything then.
284304
const coveredBy = overrides.size ? undefined : covered;
285305
const redundant = (key: string, value: number) => {
@@ -297,15 +317,21 @@ export function retainLocalRead(
297317
const protectedKey = (key: string) =>
298318
overrides.size > 0 && !key.startsWith("msg:");
299319
// Keep inherited floors first (a subset of the already bounded reserve), then
300-
// broad receipts before messages. Event age breaks ties within each scope.
301-
// Journal recency still controls newly read old history and sync.
320+
// the newest marks, so a drop raises the floor as little as it can.
321+
// Journal recency still controls newly read old history.
322+
let raised = floor;
323+
let full = false;
302324
for (const [key, value] of [...frontiers].sort(
303325
([a, av], [b, bv]) =>
304326
Number(protectedKey(b)) - Number(protectedKey(a)) ||
305-
scope(a) - scope(b) ||
306327
bv - av ||
307328
a.localeCompare(b),
308329
)) {
330+
// Once one mark does not fit, every later one is dropped too.
331+
if (full) {
332+
raised = Math.max(raised, value);
333+
continue;
334+
}
309335
if (redundant(key, value)) continue;
310336
const cost =
311337
encoder.encode(JSON.stringify(key)).byteLength +
@@ -315,11 +341,18 @@ export function retainLocalRead(
315341
if (
316342
entries.length >= READ_RESERVE_KEYS ||
317343
bytes + cost > READ_RESERVE_BYTES
318-
)
319-
break;
344+
) {
345+
full = true;
346+
raised = Math.max(raised, value);
347+
continue;
348+
}
320349
entries.push([key, value]);
321350
accepted.set(key, value);
322351
bytes += cost;
323352
}
324-
return { ...kept, reserve: Object.freeze(Object.fromEntries(entries)) };
353+
return {
354+
...kept,
355+
reserve: Object.freeze(Object.fromEntries(entries)),
356+
floor: raised,
357+
};
325358
}

‎src/features/relay/read-state-storage.ts‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@ export type ReadJournal = Readonly<{
2828
recent?: Readonly<Record<string, number>>;
2929
/** Local-only evicted receipts, deleted with this partition even by old builds. */
3030
reserve?: Readonly<Record<string, number>>;
31+
/** Local-only: every message at or before this time counts as read here.
32+
* Raised to the time of the newest mark that the reserve had no room for. */
33+
floor?: number;
3134
/** Local-only: the channel of each message that a kept mark names (see
3235
* `markMessage`), taken from a signed event. A mark does not carry its
3336
* channel, so this lets a channel mark cover it while the message is not
@@ -160,6 +163,8 @@ export function readJournal(raw: unknown, viewer: string): ReadJournal {
160163
))
161164
)
162165
throw new Error("Invalid saved own read state");
166+
if (raw.floor !== undefined && !uint32(raw.floor))
167+
throw new Error("Invalid saved read floor");
163168
const spill = raw.spill ?? [];
164169
if (
165170
!Array.isArray(spill) ||
@@ -223,6 +228,7 @@ export function readJournal(raw: unknown, viewer: string): ReadJournal {
223228
...(raw.reserve as Record<string, number> | undefined),
224229
}),
225230
homes: Object.freeze(homes),
231+
...(raw.floor ? { floor: raw.floor } : {}),
226232
own: Object.freeze({ ...own }),
227233
spill: Object.freeze([...(spill as string[])]),
228234
revision: raw.revision as number,
@@ -271,6 +277,10 @@ export function browserReadStateStorage(
271277
): ReadStateStorage {
272278
let database: Promise<IDBDatabase> | undefined;
273279
let closed = false;
280+
// The record this handle last wrote, already validated, under a random
281+
// token saved with it. Another window or build writes without the token,
282+
// so finding it again proves the stored record is this one.
283+
let written: { token: string; journal: ReadJournal } | undefined;
274284
function open() {
275285
if (closed) return Promise.reject(new Error("Read-state storage closed"));
276286
database ??= openDatabase();
@@ -290,20 +300,27 @@ export function browserReadStateStorage(
290300
let error: unknown;
291301
request.onsuccess = () => {
292302
try {
303+
const stored: unknown = request.result;
293304
const current =
294-
request.result === undefined
305+
stored === undefined
295306
? undefined
296-
: readJournal(request.result, viewer);
307+
: written && record(stored) && stored.written === written.token
308+
? written.journal
309+
: readJournal(stored, viewer);
297310
result = readJournal(change(current), viewer);
298-
store.put(result, scope);
311+
const token = crypto.randomUUID();
312+
store.put({ ...result, written: token }, scope);
313+
written = { token, journal: result };
299314
} catch (cause) {
300315
error = cause;
301316
tx.abort();
302317
}
303318
};
304319
tx.oncomplete = () => resolve(result);
305-
tx.onabort = () =>
320+
tx.onabort = () => {
321+
written = undefined;
306322
reject(error ?? tx.error ?? new Error("Read-state save aborted"));
323+
};
307324
tx.onerror = () => reject(tx.error);
308325
});
309326
},

‎src/features/relay/read-state.test.ts‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -843,6 +843,37 @@ describe("durable read-state owner", () => {
843843
expect(restarted.state().frontiers[old]).toBe(1);
844844
expect(restarted.state().frontiers["msg:unseen"]).toBeUndefined();
845845
});
846+
it("raises a local floor over marks the reserve drops, keeps it after restart and never publishes it", async () => {
847+
const f = fixture();
848+
const owner = f.make();
849+
const old = `msg:${"a".repeat(64)}`;
850+
await owner.ready;
851+
await owner.read(old, 1, () => true);
852+
const marks = Array.from({ length: 7000 }, (_, n) => ({
853+
key: `msg:${n.toString(16).padStart(64, "0")}`,
854+
timestamp: 10 + n,
855+
channelId: "elsewhere",
856+
}));
857+
await owner.readMessages(marks, undefined, () => true);
858+
const floor = owner.floor();
859+
expect(floor).toBeGreaterThan(10);
860+
expect(owner.state().frontiers[old]).toBeUndefined();
861+
for (const { key, timestamp } of [{ key: old, timestamp: 1 }, ...marks])
862+
expect(
863+
(owner.state().frontiers[key] ?? -1) >= timestamp || timestamp <= floor,
864+
).toBe(true);
865+
expect(owner.state().frontiers[marks.at(-1)?.key ?? ""]).toBe(7009);
866+
// A channel mark below the local floor still publishes: publication keeps
867+
// only the retention window.
868+
await owner.read("room", 5, () => true);
869+
expect(owner.state().frontiers.room).toBe(5);
870+
await owner.flush();
871+
expect(f.published()).toEqual({ room: 5 });
872+
owner.dispose();
873+
const restarted = f.make();
874+
await restarted.ready;
875+
expect(restarted.floor()).toBe(floor);
876+
});
846877
it("preserves the reserve through remote replay, local unread and concurrent mutations", async () => {
847878
const f = fixture();
848879
f.setJournal({ ...newReadJournal(), reserve: { "msg:old": 20 } });
@@ -904,6 +935,13 @@ describe("durable read-state owner", () => {
904935
});
905936
it("validates optional reserve data without replacing corrupt saved intent", () => {
906937
const f = fixture();
938+
expect(
939+
readJournal({ ...newReadJournal(), floor: 7 }, f.key.pubkey).floor,
940+
).toBe(7);
941+
for (const floor of [-1, 1.5, "7"])
942+
expect(() =>
943+
readJournal({ ...newReadJournal(), floor }, f.key.pubkey),
944+
).toThrow("floor");
907945
expect(readJournal(newReadJournal(), f.key.pubkey).reserve).toEqual({});
908946
for (const reserve of [
909947
[],

0 commit comments

Comments
 (0)