Skip to content

Commit 8c06965

Browse files
authored
fix(daemon): cap events.ndjson with cursor-safe rotation (#1867)
* fix(daemon): cap events.ndjson with cursor-safe rotation Rotate events.ndjson to events.ndjson.1 once it reaches AGENT_DEVICE_EVENT_LOG_MAX_BYTES (default 5 MB), keeping one rotated generation. Cursors stay absolute across rotation through a sidecar window offset, so a persisted nextCursor still names the same event; a cursor older than the retained window fails with COMMAND_FAILED and details.reason EVENT_LOG_CURSOR_EXPIRED instead of returning a wrong page. Closes #1788 * fix(daemon): verify the events.ndjson window against the files on disk Rotation recorded only a dropped-line offset, written after the rename, so a reader landing in that window mapped every absolute cursor a whole generation too far (reproduced: 5 of 58 reads returned event 17 for cursor 9), and a missing rotated file or stale sidecar shifted cursors permanently and silently. The sidecar now records each retained generation's first absolute line index, line count, and first-line digest, and is written before the rename it describes. The reader identifies each file on disk by digest, derives its start from the matching record, and checks the recorded line count and generation contiguity; anything unverifiable raises a typed EVENT_LOG_WINDOW_UNVERIFIED instead of a guessed offset. A torn snapshot (rotation landing mid-read from the threadpool) is retried, not interpreted. A corrupt sidecar fails reads typed and never blocks appends, and rotation no longer does synchronous whole-file I/O. * refactor(daemon): split event-log window placement and share one line splitter
1 parent 6984a1e commit 8c06965

9 files changed

Lines changed: 773 additions & 76 deletions

File tree

‎docs/adr/0018-unified-event-journal.md‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -199,8 +199,13 @@ statically enumerable, greppable, no dynamic subscription API. Semantics, normat
199199
stderr fallback), unchanged behavior including the `liveWrittenEventCount` flush watermark;
200200
- **session event log sink** — consumes the `session-lifecycle` trait kinds and writes today's
201201
`events.ndjson` v1 entries, reusing the existing presentation builders (`buildActionSummary`,
202-
`buildRequestSuccessEventPresentation`) and write queue; receives events from request scopes and
203-
teardown scopes alike;
202+
`buildRequestSuccessEventPresentation`), write queue, and retention window
203+
(`session-event-log-window.ts`, #1788: size-capped rotation to one retained generation,
204+
`events.ndjson.1`). Entry bytes are unchanged by the cap, but the window's sidecar
205+
(`events.ndjson.window.json`) is **new journal-owned state that cursor identity depends on**:
206+
it records each retained generation's first absolute line index, line count, and first-line
207+
digest, and the reader verifies those against the files before answering. A sink that ever owns
208+
this file owns that contract too — it is not a cache and cannot be regenerated from the entries;
204209
- **replay trace sink** — consumes `replay-trace` trait kinds, maps internal kinds to the legacy
205210
`type` values, writes the per-attempt file via bound routing context; replaces both existing
206211
append helpers;
@@ -334,13 +339,18 @@ gets built.
334339
- Out-of-request coverage: an idle-reap/shutdown `finalizeRepairTeardown` still lands its
335340
synthesized `close` as an `action.recorded` entry in `events.ndjson`, via a teardown scope, with
336341
no request active.
342+
- Retention: the size cap (#1788) is orthogonal to sink ownership but shares the file. A sink
343+
migration keeps rotation inside the one serialized write path, keeps the sidecar written before
344+
the rename it describes, and keeps reads failing typed
345+
(`EVENT_LOG_CURSOR_EXPIRED` / `EVENT_LOG_WINDOW_UNVERIFIED`) rather than answering a cursor it
346+
cannot place.
337347
- Per-attempt routing: a multi-attempt `test` run writes each attempt's trace to its own file with
338348
legacy `type` values; trace-kind events emitted with no bound destination are dropped, not
339349
misrouted. A **concurrent nested-action regression** runs sharded attempts in parallel — each
340350
performing nested dispatch that rebinds `session`/`logPath` via `createRequestExecutionScope` —
341351
and proves per-fork isolation across **all three routed outputs**: each `replay-timing.ndjson`,
342-
each per-request diagnostics ndjson, and each session's `events.ndjson` contain only their own
343-
shard's events — no cross-writes, no drops — including events emitted after `await` points.
352+
each per-request diagnostics ndjson, and each session's `events.ndjson` plus its retained generation contain only
353+
their own shard's events — no cross-writes, no drops — including events emitted after `await` points.
344354
- Sink isolation: a sink that throws does not affect the buffer, other sinks, or the response;
345355
ordering across sinks is registration order.
346356
- `cost.runnerRoundTrips` parity: the trait-derived set equals the current literal list; the
@@ -368,7 +378,8 @@ Each step lands green and independently useful:
368378
3. **Session event log as sink + teardown scopes** — request-lifecycle emit points route through
369379
the journal; the `events.ndjson` writer becomes a sink; idle-reap/shutdown finalizers open
370380
session-scoped teardown scopes so out-of-request `action.recorded` events keep flowing; the
371-
`events` command and pagination untouched.
381+
`events` command keeps its paging contract, including the typed cursor-expiry and
382+
window-verification errors the #1788 retention window added to it.
372383
4. **Replay trace as sink** — the journal fork primitive lands with its concurrent-shard
373384
regression; both trace helpers are replaced by one sink with fork-bound per-attempt routing and
374385
legacy `type` mapping; the declared redaction change ships here with its fixture and changelog
Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
import { afterEach, test } from 'vitest';
2+
import assert from 'node:assert/strict';
3+
import fs from 'node:fs';
4+
import path from 'node:path';
5+
import { AppError } from '@agent-device/kernel/errors';
6+
import { mkdtempForTestSync } from '../../__tests__/test-utils/tmp-dir.ts';
7+
import {
8+
appendSessionEvent,
9+
flushSessionEventLogWrites,
10+
readSessionEventLog,
11+
resolveSessionEventLogPath,
12+
} from '../session-event-log.ts';
13+
import { digestEventLogLine } from '../session-event-log-lines.ts';
14+
import {
15+
EVENT_LOG_CURSOR_EXPIRED_REASON,
16+
EVENT_LOG_WINDOW_UNVERIFIED_REASON,
17+
resolveRotatedSessionEventLogPath,
18+
resolveSessionEventLogMaxBytes,
19+
} from '../session-event-log-window.ts';
20+
21+
const originalMaxBytes = process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES;
22+
23+
afterEach(() => {
24+
if (originalMaxBytes === undefined) delete process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES;
25+
else process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES = originalMaxBytes;
26+
});
27+
28+
async function appendEvents(eventLogPath: string, from: number, to: number): Promise<void> {
29+
for (let index = from; index < to; index += 1) {
30+
appendSessionEvent(eventLogPath, 'default', {
31+
kind: 'request.started',
32+
command: 'snapshot',
33+
summary: `event ${index}`,
34+
});
35+
}
36+
await flushSessionEventLogWrites(eventLogPath);
37+
}
38+
39+
function summaries(eventLogPath: string, cursor?: string, limit?: number): string[] {
40+
return readSessionEventLog(eventLogPath, { cursor, limit }).events.map(
41+
(event) => event.summary ?? '',
42+
);
43+
}
44+
45+
function readError(eventLogPath: string, cursor: string): AppError {
46+
try {
47+
readSessionEventLog(eventLogPath, { cursor });
48+
} catch (error) {
49+
assert.ok(error instanceof AppError, `expected an AppError, got ${String(error)}`);
50+
return error;
51+
}
52+
throw new assert.AssertionError({ message: `expected cursor ${cursor} to be refused` });
53+
}
54+
55+
function retainedLineCount(eventLogPath: string): number {
56+
return [resolveRotatedSessionEventLogPath(eventLogPath), eventLogPath]
57+
.filter((file) => fs.existsSync(file))
58+
.map((file) => fs.readFileSync(file, 'utf8').split('\n').filter(Boolean).length)
59+
.reduce((sum, count) => sum + count, 0);
60+
}
61+
62+
test('rotation caps events.ndjson while cursors stay absolute across the retained window', async () => {
63+
const eventLogPath = resolveSessionEventLogPath(
64+
mkdtempForTestSync('agent-device-event-log-cap-'),
65+
);
66+
// One entry is ~110 bytes; a 1 KB cap rotates every 9-10 entries, so 40
67+
// entries force several rotations and at least one dropped generation.
68+
process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES = '1024';
69+
assert.equal(resolveSessionEventLogMaxBytes(), 1024);
70+
71+
await appendEvents(eventLogPath, 0, 40);
72+
73+
assert.equal(fs.existsSync(resolveRotatedSessionEventLogPath(eventLogPath)), true);
74+
assert.ok(
75+
fs.statSync(eventLogPath).size <= 1024 + 200,
76+
`active file must stay near the cap, got ${fs.statSync(eventLogPath).size} bytes`,
77+
);
78+
const retained = retainedLineCount(eventLogPath);
79+
assert.ok(retained < 40, `expected dropped events, retained ${retained}`);
80+
81+
// A cursor inside the retained window names the same event it named before
82+
// rotation, spanning the rotated and active files.
83+
const earliest = 40 - retained;
84+
assert.deepEqual(
85+
summaries(eventLogPath, String(earliest), 500),
86+
Array.from({ length: retained }, (_, index) => `event ${earliest + index}`),
87+
);
88+
const tail = readSessionEventLog(eventLogPath, { cursor: '38', limit: 1 });
89+
assert.deepEqual(
90+
tail.events.map((event) => event.summary),
91+
['event 38'],
92+
);
93+
assert.equal(tail.nextCursor, '39');
94+
assert.deepEqual(summaries(eventLogPath, tail.nextCursor, 1), ['event 39']);
95+
96+
const expired = readError(eventLogPath, '0');
97+
assert.equal(expired.code, 'COMMAND_FAILED');
98+
assert.equal(expired.details?.reason, EVENT_LOG_CURSOR_EXPIRED_REASON);
99+
assert.equal(expired.details?.earliestCursor, earliest);
100+
assert.equal(typeof expired.details?.hint, 'string');
101+
102+
// The window survives further rotations.
103+
await appendEvents(eventLogPath, 40, 60);
104+
assert.deepEqual(summaries(eventLogPath, '58', 2), ['event 58', 'event 59']);
105+
});
106+
107+
test('a hand-deleted rotated generation expires its cursors instead of shifting them', async () => {
108+
const eventLogPath = resolveSessionEventLogPath(
109+
mkdtempForTestSync('agent-device-event-log-gap-'),
110+
);
111+
process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES = '1024';
112+
await appendEvents(eventLogPath, 0, 40);
113+
114+
const retained = retainedLineCount(eventLogPath);
115+
const earliest = 40 - retained;
116+
const rotatedPath = resolveRotatedSessionEventLogPath(eventLogPath);
117+
const rotatedLines = fs.readFileSync(rotatedPath, 'utf8').split('\n').filter(Boolean).length;
118+
fs.rmSync(rotatedPath);
119+
120+
// Cursors that lived in the deleted generation are refused, not answered
121+
// with the surviving file's events; the surviving file keeps its own indices.
122+
const expired = readError(eventLogPath, String(earliest));
123+
assert.equal(expired.details?.reason, EVENT_LOG_CURSOR_EXPIRED_REASON);
124+
assert.equal(expired.details?.earliestCursor, earliest + rotatedLines);
125+
assert.deepEqual(summaries(eventLogPath, String(earliest + rotatedLines), 1), [
126+
`event ${earliest + rotatedLines}`,
127+
]);
128+
});
129+
130+
test('a reader that lands between the window write and the rename still resolves cursors', async () => {
131+
const eventLogPath = resolveSessionEventLogPath(
132+
mkdtempForTestSync('agent-device-event-log-race-'),
133+
);
134+
process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES = '1024';
135+
await appendEvents(eventLogPath, 0, 40);
136+
137+
// Rotation writes the sidecar describing the post-rename world *before* the
138+
// rename, so this exact state is observable on the normal event loop. Build
139+
// it deterministically: record the live active file as the next generation
140+
// while it is still the active file.
141+
const windowPath = `${eventLogPath}.window.json`;
142+
const sidecar = JSON.parse(fs.readFileSync(windowPath, 'utf8')) as {
143+
version: number;
144+
generations: { firstLineIndex: number; lineCount: number; firstLineDigest: string }[];
145+
};
146+
const newest = sidecar.generations[sidecar.generations.length - 1]!;
147+
const activeLines = fs.readFileSync(eventLogPath, 'utf8').split('\n').filter(Boolean);
148+
const activeStart = newest.firstLineIndex + newest.lineCount;
149+
fs.writeFileSync(
150+
windowPath,
151+
JSON.stringify({
152+
version: sidecar.version,
153+
generations: [
154+
newest,
155+
{
156+
firstLineIndex: activeStart,
157+
lineCount: activeLines.length,
158+
firstLineDigest: digestEventLogLine(activeLines[0]!),
159+
},
160+
],
161+
}),
162+
'utf8',
163+
);
164+
165+
// Both retained files must still answer for the events they actually hold.
166+
assert.deepEqual(summaries(eventLogPath, String(activeStart), 1), [`event ${activeStart}`]);
167+
assert.deepEqual(summaries(eventLogPath, String(newest.firstLineIndex), 1), [
168+
`event ${newest.firstLineIndex}`,
169+
]);
170+
});
171+
172+
test('a retained generation whose length moved is refused rather than mapped', async () => {
173+
const eventLogPath = resolveSessionEventLogPath(
174+
mkdtempForTestSync('agent-device-event-log-tamper-'),
175+
);
176+
process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES = '1024';
177+
await appendEvents(eventLogPath, 0, 40);
178+
179+
const rotatedPath = resolveRotatedSessionEventLogPath(eventLogPath);
180+
fs.appendFileSync(
181+
rotatedPath,
182+
`${JSON.stringify({ version: 1, ts: 'x', session: 'd', kind: 'action.recorded' })}\n`,
183+
);
184+
185+
const unverified = readError(eventLogPath, '0');
186+
assert.equal(unverified.details?.reason, EVENT_LOG_WINDOW_UNVERIFIED_REASON);
187+
});
188+
189+
test('an unreadable window sidecar fails reads typed and never blocks appends', async () => {
190+
const eventLogPath = resolveSessionEventLogPath(
191+
mkdtempForTestSync('agent-device-event-log-bad-'),
192+
);
193+
process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES = '1024';
194+
await appendEvents(eventLogPath, 0, 40);
195+
196+
fs.writeFileSync(`${eventLogPath}.window.json`, '{ not json', 'utf8');
197+
198+
const unverified = readError(eventLogPath, '0');
199+
assert.equal(unverified.code, 'COMMAND_FAILED');
200+
assert.equal(unverified.details?.reason, EVENT_LOG_WINDOW_UNVERIFIED_REASON);
201+
assert.equal(typeof unverified.details?.hint, 'string');
202+
203+
// Writes keep flowing past the corrupt sidecar, including across the
204+
// rotations the following appends trigger.
205+
const sizeBefore = fs.statSync(eventLogPath).size;
206+
await appendEvents(eventLogPath, 40, 60);
207+
assert.ok(
208+
fs.statSync(eventLogPath).size > 0 && retainedLineCount(eventLogPath) >= 9,
209+
`appends must survive a corrupt sidecar (size before ${sizeBefore})`,
210+
);
211+
assert.equal(readError(eventLogPath, '0').details?.reason, EVENT_LOG_WINDOW_UNVERIFIED_REASON);
212+
});
213+
214+
test('a rotated generation with no window record is refused rather than renumbered', async () => {
215+
const dir = mkdtempForTestSync('agent-device-event-log-orphan-');
216+
const eventLogPath = resolveSessionEventLogPath(dir);
217+
await appendEvents(eventLogPath, 0, 3);
218+
fs.copyFileSync(eventLogPath, resolveRotatedSessionEventLogPath(eventLogPath));
219+
220+
assert.equal(readError(eventLogPath, '0').details?.reason, EVENT_LOG_WINDOW_UNVERIFIED_REASON);
221+
});
222+
223+
test('the default cap leaves an ordinary session log unrotated, and a non-integer override is ignored', async () => {
224+
const dir = mkdtempForTestSync('agent-device-event-log-default-cap-');
225+
const eventLogPath = resolveSessionEventLogPath(dir);
226+
// "5MB" must not be read as 5 bytes, which would rotate on every entry.
227+
process.env.AGENT_DEVICE_EVENT_LOG_MAX_BYTES = '5MB';
228+
assert.equal(resolveSessionEventLogMaxBytes(), 5 * 1024 * 1024);
229+
230+
await appendEvents(eventLogPath, 0, 3);
231+
232+
assert.deepEqual(fs.readdirSync(dir), [path.basename(eventLogPath)]);
233+
assert.deepEqual(summaries(eventLogPath), ['event 0', 'event 1', 'event 2']);
234+
});

0 commit comments

Comments
 (0)