Skip to content

Commit 9509f6b

Browse files
committed
fix(daemon): give each startup its own marker file and retire it at its owner
The integration leak oracle failed because every daemon appended its startup phase markers to one shared `daemon-startup-phases.jsonl` beside the registration and nothing retired that file, so any completed launch left state-dir residue the strict oracle names as a stray entry. Ownership now follows the file name: each launch writes only to `daemon-startup-phases-<pid>.jsonl` through `resolveDaemonStartupPhasesPath(stateDir, pid)`. The writing daemon removes its file once it has registered; the launching client removes it once that process's exit is joined (in `retireStartupAttempt`, after the exit is observed, never before); and each new registration-lock owner sweeps the files of provably dead pids next to the shutdown-report removal. A live pid's file is always kept, so a concurrent or replaced launch can neither add to, hide behind, nor delete another launch's evidence, and the client no longer needs `spawnedAtMs` to attribute markers to its own launch. Along the protocol boundary: - `recordDaemonStartupMarker` takes an optional `reportFailure` callback so producers surface a refused append on stderr (the daemon log tail the waiting client already ships as failure evidence) instead of swallowing it; the append stays non-fatal and a crash-truncated final line is separated so it cannot glue two records together. - The phase list gains real completed boundaries, `composition-loaded` and `recovery-finished`, and no heartbeats, so silence still means a stalled startup and progress extends the stall window exactly as far as the work took. - The launch-exit join drops its redundant `.catch(() => {})`: the `runCmdDetachedMonitored` owner resolves the promise on error and exit, so it never rejects. - Startup code splits out of `daemon-client-lifecycle.ts` into `daemon-client-startup.ts` (budget, wait loop, marker refresh, hints), keeping both modules under the 1000-line tripwire; importers attach and ensure point at the owning module directly. Regression coverage: the budget suite now pins per-launch attribution, including a fresh daemon producing its own markers while an earlier launch's leftover file is swept at the new registration, and a new protocol suite pins the per-pid paths, the dead-pid sweep, removal idempotence, and the write-failure seam. Refs #3782, #3791
1 parent 8058ef2 commit 9509f6b

10 files changed

Lines changed: 1156 additions & 771 deletions
Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
import { appendFile, readFile, writeFile } from 'node:fs/promises';
2+
import { spawnSync } from 'node:child_process';
3+
import { existsSync, mkdirSync, readFileSync } from 'node:fs';
4+
import path from 'node:path';
5+
import { describe, expect, it } from 'vitest';
6+
import {
7+
readDaemonStartupPhaseMarkers,
8+
recordDaemonStartupMarker,
9+
removeDaemonStartupPhasesFile,
10+
resolveDaemonStartupPhasesPath,
11+
tryAcquireDaemonRegistration,
12+
} from '@agent-device/daemon-contracts/daemon-registration-owner';
13+
import { resolveDaemonPaths } from '@agent-device/daemon-contracts/daemon-resolution';
14+
import { mkdtempForTest } from '@agent-device/test-kit';
15+
16+
describe('per-launch startup marker files', () => {
17+
it('scopes marker files to the writing pid so a fresh launch never reads an older launch phases', async () => {
18+
const stateDir = await mkdtempForTest('agent-device-markers-');
19+
const first = resolveDaemonStartupPhasesPath(stateDir, 1111);
20+
const second = resolveDaemonStartupPhasesPath(stateDir, 2222);
21+
22+
recordDaemonStartupMarker(first, 'lock-acquired');
23+
24+
expect(second).not.toBe(first);
25+
expect(existsSync(second)).toBe(false);
26+
27+
recordDaemonStartupMarker(second, 'modules-loaded');
28+
const firstPhases = readDaemonStartupPhaseMarkers(readFileSync(first, 'utf8'));
29+
const secondPhases = readDaemonStartupPhaseMarkers(readFileSync(second, 'utf8'));
30+
expect(Object.keys(firstPhases)).toEqual(['lock-acquired']);
31+
expect(Object.keys(secondPhases)).toEqual(['modules-loaded']);
32+
});
33+
34+
it('keeps the earliest record when a replayed phase is appended after a later one', () => {
35+
const text = [
36+
'{"data":{"startupPhase":"listening","timestampMs":500}}',
37+
'{"data":{"startupPhase":"lock-acquired","timestampMs":20}}',
38+
'{"data":{"startupPhase":"lock-acquired","timestampMs":35}}',
39+
].join('\n');
40+
const phases = readDaemonStartupPhaseMarkers(text);
41+
expect(phases['lock-acquired']).toBe(20);
42+
expect(phases.listening).toBe(500);
43+
});
44+
45+
it('ignores unknown phases and unparsable lines so the protocol stays additive', () => {
46+
const text = [
47+
'{"data":{"startupPhase":"heartbeat","timestampMs":10}}',
48+
'{"data":{"startupPhase":"registered","timestampMs":"7"}}',
49+
'not json',
50+
'{"data":{"startupPhase":"listening","timestampMs":30}}',
51+
].join('\n');
52+
expect(readDaemonStartupPhaseMarkers(text)).toEqual({ listening: 30 });
53+
});
54+
});
55+
56+
describe('registration-time sweep', () => {
57+
it('retires marker files of provably-dead pids and keeps files of live pids', async () => {
58+
const paths = resolveDaemonPaths(await mkdtempForTest('agent-device-markers-'));
59+
const deadPid = spawnDeadPid();
60+
const own = resolveDaemonStartupPhasesPath(paths.baseDir, process.pid);
61+
const dead = resolveDaemonStartupPhasesPath(paths.baseDir, deadPid);
62+
const legacy = path.join(paths.baseDir, 'daemon-startup-phases.jsonl');
63+
await Promise.all([
64+
writeFile(own, 'x\n', 'utf8'),
65+
writeFile(dead, 'x\n', 'utf8'),
66+
writeFile(legacy, 'x\n', 'utf8'),
67+
]);
68+
69+
const attempt = await tryAcquireDaemonRegistration(paths);
70+
expect(attempt.status).toBe('acquired');
71+
if (attempt.status !== 'acquired') throw new Error('registration refused');
72+
73+
expect(existsSync(dead)).toBe(false);
74+
expect(existsSync(own)).toBe(true);
75+
// The sweep retires only its own kind: an unknown entry stays as evidence.
76+
expect(existsSync(legacy)).toBe(true);
77+
await attempt.owner.finish();
78+
});
79+
80+
it('retires files whose pid is unparseable', async () => {
81+
const paths = resolveDaemonPaths(await mkdtempForTest('agent-device-markers-'));
82+
const stale = path.join(paths.baseDir, 'daemon-startup-phases-.jsonl');
83+
await writeFile(stale, 'x\n', 'utf8');
84+
85+
const attempt = await tryAcquireDaemonRegistration(paths);
86+
expect(attempt.status).toBe('acquired');
87+
if (attempt.status !== 'acquired') throw new Error('registration refused');
88+
89+
expect(existsSync(stale)).toBe(false);
90+
await attempt.owner.finish();
91+
});
92+
});
93+
94+
describe('removeDaemonStartupPhasesFile', () => {
95+
it('removes a marker file and tolerates a file already gone', async () => {
96+
const stateDir = await mkdtempForTest('agent-device-markers-');
97+
const file = resolveDaemonStartupPhasesPath(stateDir, 4242);
98+
await writeFile(file, 'x\n', 'utf8');
99+
100+
removeDaemonStartupPhasesFile(file);
101+
expect(existsSync(file)).toBe(false);
102+
103+
expect(() => removeDaemonStartupPhasesFile(file)).not.toThrow();
104+
});
105+
});
106+
107+
describe('recordDaemonStartupMarker failure seam', () => {
108+
it('reports a write failure through the callback instead of throwing, keeping startup running', async () => {
109+
const stateDir = await mkdtempForTest('agent-device-markers-');
110+
// A directory in place of the marker file makes every append fail.
111+
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 777);
112+
mkdirSync(markerPath, { recursive: true });
113+
114+
const reported: unknown[] = [];
115+
expect(() =>
116+
recordDaemonStartupMarker(markerPath, 'modules-loaded', (error) => {
117+
reported.push(error);
118+
}),
119+
).not.toThrow();
120+
121+
expect(reported).toHaveLength(1);
122+
expect(reported[0]).toHaveProperty('message');
123+
});
124+
125+
it('keeps appending valid JSONL when an incomplete line precedes a crash-replay append', async () => {
126+
const stateDir = await mkdtempForTest('agent-device-markers-');
127+
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 999);
128+
await appendFile(markerPath, '{"data":{"startupPhase":"modules-lo', 'utf8');
129+
130+
recordDaemonStartupMarker(markerPath, 'composition-loaded');
131+
132+
const tail = await readFile(markerPath, 'utf8');
133+
expect(Object.keys(readDaemonStartupPhaseMarkers(tail))).toEqual(['composition-loaded']);
134+
});
135+
});
136+
137+
function spawnDeadPid(): number {
138+
// A process that has already exited proves the liveness check rejects it,
139+
// the same way the daemon-process takeover guards prove it.
140+
const child = spawnSync(process.execPath, ['-e', ''], { encoding: 'utf8' });
141+
if (child.error) throw child.error;
142+
const pid = child.pid;
143+
if (pid === undefined) throw new Error('spawnSync returned no pid');
144+
return pid;
145+
}

‎packages/daemon-contracts/src/daemon-registration-owner.ts‎

Lines changed: 93 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import {
99
type DaemonTerminationResult,
1010
} from './daemon-process.ts';
1111
import {
12+
isProcessAlive,
1213
readCurrentOwnerIdentity,
1314
readProcessStartTime,
1415
type OwnerIdentity,
@@ -47,16 +48,24 @@ import {
4748
export const DAEMON_STARTUP_EXIT_CODES = Object.freeze({ busy: 75, unproven: 78 });
4849

4950
/**
50-
* Startup phase markers (#3782): the record a daemon leaves beside its registration of how far its
51+
* Startup phase markers (#3782): the record a daemon writes beside its registration of how far its
5152
* startup got. The client waiting for registration reads this record to tell "slow but progressing"
52-
* from "stuck". Each marker is one JSONL line appended to a dedicated state-directory file, so no
53-
* `daemon.log` reader has to learn a new line shape and `daemon.log` keeps its existing ownership.
54-
* The record is append-only across startups: a waiting client scopes it to its own launch by the
55-
* marker timestamps, never by deleting what an earlier startup left.
53+
* from "stuck". One launch owns exactly one file, named for the pid that wrote it: an earlier or a
54+
* concurrent launch cannot add to, replace, or hide another launch's markers, and neither the file
55+
* nor a waiting client ever needs a timestamp to decide which startup a marker belongs to. A phase
56+
* is recorded when the work it names has finished; the record never carries a heartbeat, so silence
57+
* stays what a stalled startup looks like.
58+
*
59+
* Ownership, so the record never becomes residue: the writer removes the file once its startup
60+
* completes; the client that launched a process removes it once that process's exit is joined; and
61+
* each new registration owner sweeps the files of provably dead pids when it takes the lock. Each
62+
* marker is one JSONL line appended to the launch's own file.
5663
*/
5764
const DAEMON_STARTUP_PHASES = Object.freeze([
5865
'modules-loaded',
66+
'composition-loaded',
5967
'lock-acquired',
68+
'recovery-finished',
6069
'listening',
6170
'registered',
6271
] as const);
@@ -65,35 +74,96 @@ export type DaemonStartupPhase = (typeof DAEMON_STARTUP_PHASES)[number];
6574

6675
const DAEMON_STARTUP_PHASE_SET: ReadonlySet<string> = new Set<string>(DAEMON_STARTUP_PHASES);
6776

68-
const DAEMON_STARTUP_PHASES_FILE = 'daemon-startup-phases.jsonl';
77+
const DAEMON_STARTUP_PHASES_ENTRY = /^daemon-startup-phases-(\d*)\.jsonl$/;
6978

70-
type DaemonStartupMarkerRecord = {
71-
data?: { startupPhase?: unknown; timestampMs?: unknown };
72-
};
73-
74-
export function resolveDaemonStartupPhasesPath(stateDir: string): string {
75-
return path.join(stateDir, DAEMON_STARTUP_PHASES_FILE);
79+
/** The marker file the daemon process `pid` writes in `stateDir`; an absent file is no evidence. */
80+
export function resolveDaemonStartupPhasesPath(stateDir: string, pid: number): string {
81+
return path.join(stateDir, `daemon-startup-phases-${pid}.jsonl`);
7682
}
7783

84+
/**
85+
* Appends one phase marker for the launch owning `startupPhasesPath`. A marker is progress
86+
* evidence, never a startup requirement: a host that refuses the append must not lose a daemon
87+
* that can still register. Producers that own a diagnostics channel pass `reportFailure` so the
88+
* refusal is observable where the waiting client already reads failure evidence (the daemon log
89+
* tail in the startup failure details); a client never receives a false marker from a broken
90+
* channel — it reads a missing record as no evidence.
91+
*/
7892
export function recordDaemonStartupMarker(
7993
startupPhasesPath: string,
8094
phase: DaemonStartupPhase,
95+
reportFailure?: (error: NormalizedError) => void,
8196
): void {
8297
try {
8398
const timestampMs = Date.now();
8499
fs.mkdirSync(path.dirname(startupPhasesPath), { recursive: true });
100+
// A crash mid-append can leave the file without a trailing newline; without a separator the
101+
// next append would glue into one unparsable line and lose both records.
102+
const separator = endsWithoutNewline(startupPhasesPath) ? '\n' : '';
85103
fs.appendFileSync(
86104
startupPhasesPath,
87-
`${JSON.stringify({ data: { startupPhase: phase, timestampMs } })}\n`,
105+
`${separator}${JSON.stringify({ data: { startupPhase: phase, timestampMs } })}\n`,
88106
);
107+
} catch (error) {
108+
reportFailure?.(normalizeError(error));
109+
}
110+
}
111+
112+
function endsWithoutNewline(file: string): boolean {
113+
let fd: number | undefined;
114+
try {
115+
const stats = fs.statSync(file);
116+
if (stats.size === 0) return false;
117+
const buffer = Buffer.alloc(1);
118+
fd = fs.openSync(file, 'r');
119+
if (fs.readSync(fd, buffer, 0, 1, stats.size - 1) !== 1) return false;
120+
return buffer[0] !== 0x0a;
89121
} catch {
90-
// A marker is progress evidence, never a startup requirement: a host that refuses the append
91-
// must not lose a daemon that can still register.
122+
return false;
123+
} finally {
124+
if (fd !== undefined) fs.closeSync(fd);
92125
}
93126
}
94127

95128
/**
96-
* The phases observed in a startup marker tail, keyed by phase, holding each phase's earliest
129+
* Removes one startup's marker file. The writer calls it once its own startup completes; the client
130+
* that launched a process calls it once that process's exit is joined. Both callers can prove no
131+
* other writer can still append to this file, which is what the pid-specific name encodes.
132+
*/
133+
export function removeDaemonStartupPhasesFile(startupPhasesPath: string): void {
134+
try {
135+
fs.rmSync(startupPhasesPath, { force: true });
136+
} catch {
137+
// Evidence removal never fails a daemon that is already up or a client that already joined.
138+
}
139+
}
140+
141+
/**
142+
* Sweeps marker files left by provably dead launches, from the position of a process that holds the
143+
* registration lock. A live pid's file is kept: its launch may still be between phases, and only
144+
* that launch's own completion or a joined exit may remove it. A file whose name carries no usable
145+
* pid cannot be owned by any launch, so this owner retires it too.
146+
*/
147+
function sweepDeadDaemonStartupPhasesFiles(stateDir: string): void {
148+
let entries: string[];
149+
try {
150+
entries = fs.readdirSync(stateDir);
151+
} catch {
152+
return;
153+
}
154+
for (const entry of entries) {
155+
const match = DAEMON_STARTUP_PHASES_ENTRY.exec(entry);
156+
if (match === null) continue;
157+
const deadPid = Number(match[1]);
158+
if (deadPid === process.pid) continue;
159+
if (!Number.isSafeInteger(deadPid) || deadPid <= 0 || !isProcessAlive(deadPid)) {
160+
removeDaemonStartupPhasesFile(path.join(stateDir, entry));
161+
}
162+
}
163+
}
164+
165+
/**
166+
* The phases observed in one launch's marker tail, keyed by phase, holding each phase's earliest
97167
* occurrence. Unparsable and unknown lines are ignored: the protocol is additive, and a client must
98168
* never fail a startup over a line shape it does not know.
99169
*/
@@ -124,6 +194,10 @@ export function readDaemonStartupPhaseMarkers(
124194
}
125195
return phases;
126196
}
197+
198+
type DaemonStartupMarkerRecord = {
199+
data?: { startupPhase?: unknown; timestampMs?: unknown };
200+
};
127201
export type DaemonRegistrationFields = Readonly<{
128202
socketPort?: number;
129203
httpPort?: number;
@@ -163,6 +237,9 @@ export async function tryAcquireDaemonRegistration(
163237
try {
164238
acquisition.assertHeld();
165239
fs.rmSync(resolveDaemonShutdownReportPath(boundPaths.baseDir), { force: true });
240+
// A dead launch's markers are this owner's to sweep: only the launch itself or a client that
241+
// joined its exit may remove a live launch's file, and neither can still reach this one.
242+
sweepDeadDaemonStartupPhasesFiles(boundPaths.baseDir);
166243
} catch (error) {
167244
await releaseRegistrationAfterFailure(acquisition, error, boundPaths.logPath);
168245
}

‎packages/daemon/src/server/http/daemon-runtime.ts‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ import {
8080
tryAcquireDaemonRegistration,
8181
DAEMON_STARTUP_EXIT_CODES,
8282
recordDaemonStartupMarker,
83+
removeDaemonStartupPhasesFile,
8384
resolveDaemonStartupPhasesPath,
8485
type DaemonRegistrationOwner,
8586
} from '@agent-device/daemon-contracts/daemon-registration-owner';
@@ -778,8 +779,16 @@ export async function startDaemonRuntime(
778779
}
779780
const registration = acquisition.owner;
780781
// #3782: the client waiting on this state dir reads these markers to tell a slow daemon from a
781-
// stuck one.
782-
recordDaemonStartupMarker(resolveDaemonStartupPhasesPath(baseDir), 'lock-acquired');
782+
// stuck one. Each marker names a phase whose work has finished; this launch's file is named for
783+
// its own pid, so no other launch's evidence can add to or hide it.
784+
const startupPhasesPath = resolveDaemonStartupPhasesPath(baseDir, process.pid);
785+
const recordStartupPhase = (phase: Parameters<typeof recordDaemonStartupMarker>[1]): void =>
786+
recordDaemonStartupMarker(startupPhasesPath, phase, (error) => {
787+
// Same contract as the entry's markers: the refusal goes where the waiting client already
788+
// reads failure evidence (the daemon log), and startup keeps going.
789+
stderr.write(`Daemon startup marker write failed: ${error.message}\n`);
790+
});
791+
recordStartupPhase('lock-acquired');
783792

784793
let servers: DaemonServer[] = [];
785794
let socketPort: number | undefined;
@@ -835,15 +844,22 @@ export async function startDaemonRuntime(
835844
data: { error: error instanceof Error ? error.message : String(error) },
836845
});
837846
});
847+
// #3782: the recovery chain above (owner-lifecycle configuration, legacy-marker and app-log
848+
// recovery, owned-process reaping, web-orphan cleanup) is real completed work: its boundary
849+
// extends the stall window exactly as much as it took, never more.
850+
recordStartupPhase('recovery-finished');
838851
const opened = await openDaemonServers();
839852
servers = opened.servers;
840853
socketPort = opened.socketPort;
841854
httpPort = opened.httpPort;
842855
// #3782: the last two progress markers. A client that polls the registration and the markers
843856
// together can tell "listening but slow to answer" from "never progressed".
844-
recordDaemonStartupMarker(resolveDaemonStartupPhasesPath(baseDir), 'listening');
857+
recordStartupPhase('listening');
845858
publishDaemonInfo(socketPort, httpPort);
846-
recordDaemonStartupMarker(resolveDaemonStartupPhasesPath(baseDir), 'registered');
859+
recordStartupPhase('registered');
860+
// This startup's evidence has done its job: no waiting client needs it anymore, and this
861+
// process is the only writer of the file, so this is the removal its owner contract allows.
862+
removeDaemonStartupPhasesFile(startupPhasesPath);
847863
stopMetadataLossWatch = armDaemonMetadataLossWatch(baseDir, infoPath, logPath, daemonIdentity);
848864
await flushDaemonStartupDiagnostics(logPath, startupDiagnostics);
849865
for (const plugin of deps.pluginInventory.filter(({ state }) => state === 'failed')) {

‎src/__tests__/test-utils/registered-daemon-fixture.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ export function registeredDaemonFixtureArgs(
3434
entry,
3535
`import fs from 'node:fs';
3636
import path from 'node:path';
37-
import { DAEMON_STARTUP_EXIT_CODES, tryAcquireDaemonRegistration } from ${JSON.stringify(registrationUrl)};
37+
import { DAEMON_STARTUP_EXIT_CODES, tryAcquireDaemonRegistration, recordDaemonStartupMarker, resolveDaemonStartupPhasesPath } from ${JSON.stringify(registrationUrl)};
3838
const paths = ${JSON.stringify(paths)};
3939
const barrier = ${JSON.stringify(acquisitionBarrier)};
4040
if (barrier) {
@@ -43,6 +43,10 @@ if (barrier) {
4343
}
4444
const acquired = await tryAcquireDaemonRegistration(paths);
4545
if (acquired.status !== 'acquired') process.exit(DAEMON_STARTUP_EXIT_CODES[acquired.status]);
46+
// A suite that wants this child to behave like the real entry's marker producer drops this flag
47+
// file before spawning; the marker is written through the real protocol, to this pid's own file.
48+
if (fs.existsSync(path.join(paths.baseDir, 'emit-startup-markers')))
49+
recordDaemonStartupMarker(resolveDaemonStartupPhasesPath(paths.baseDir, process.pid), 'lock-acquired');
4650
process.on('SIGTERM', async () => {
4751
if (fs.existsSync(path.join(paths.baseDir, 'ignore-sigterm'))) return;
4852
const deferred = path.join(paths.baseDir, 'repair-on-shutdown.json');

0 commit comments

Comments
 (0)