Skip to content

Commit f45bc9d

Browse files
committed
fix(daemon): report startup-ceiling when any live startup is pending at the deadline
When the launched contender exited busy while another live daemon held the registration and deferred publication, the wait loop ended in an unlabeled 'timeout' result that mapped to startup-exited with the generic hint. Any deadline with a live startup still pending, this client's launch or a verified live lock owner, now reports startup-ceiling with the effective ceiling and the overload hint; startup-exited stays reserved for a real exit or no live startup. Remove the 'timeout' wait kind so the unlabeled state cannot exist and add the sendToDaemon-boundary regressions for both outcomes. Also compress this PR's narration and duplicate test setup to keep the gross diff within one coherent change, and add the Unreleased changelog entry for the startup budget, the stall window, and the three reasons.
1 parent bef883f commit f45bc9d

12 files changed

Lines changed: 255 additions & 413 deletions

‎.github/workflows/ios.yml‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -169,10 +169,8 @@ jobs:
169169
- name: Preflight iOS runner through public CLI
170170
run: |
171171
pnpm clean:daemon
172-
# Time the cold managed startup at the public SDK seam. The clean state dir above makes
173-
# this ensure the cold start, and `started` says whether it truly started one. The CLI
174-
# `daemon_startup` timer cannot be this evidence: it wraps only attach, which runs after
175-
# the ensured transport has already finished starting the daemon.
172+
# The public-SDK timing record is the cold-startup evidence; the CLI `daemon_startup`
173+
# timer only wraps attach, after the ensured transport already started the daemon.
176174
node --input-type=module -e '
177175
import { performance } from "node:perf_hooks";
178176
const { ensureDaemon } = await import("./dist/src/index.js");

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
## Unreleased
44

5+
- A client starting a local daemon now waits while the daemon process stays alive and keeps publishing startup progress, so an overloaded host makes startup slower instead of failing it. The wait stops at a ceiling of 120 s by default; set `AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS` to a positive integer number of milliseconds to override it for the CLI, MCP, and Node clients alike. A failed startup names the outcome in `error.details.reason`, and you should branch on it, never on the message: `startup-exited` (the process ended before registering), `startup-stalled` (alive but no startup progress for a fixed 60 s window, which the ceiling variable does not change; the daemon is stopped), or `startup-ceiling` (a startup was still live — this client's daemon or another client's — when the ceiling passed; the client stops its own launch and never signals another client's live daemon). `details.phases` carries the startup phase timings the client observed and `details.startupTimeoutMs` the effective ceiling.
6+
57
- The `daemon-not-running` hint now always names `--state-dir <dir>`, and a daemon force-retired for outlasting its drain is reported as `Replaced unreachable daemon` instead of `Replaced idle daemon`. `daemon-drain-timeout` and the post-drain force retire are documented in the client API and `help`.
68
- Fix `ensureDaemon()`, the CLI and MCP killing a daemon they had just asked to leave. After the daemon answers the quiesce request with `draining`, the client now waits up to 60 s for it to exit by itself and clean up its registration, so plugin disposal and lease finalization finish (on Windows a signal ends the process outright, and on POSIX a drain longer than 3 s was killed). A daemon still running at the deadline that fails its liveness probes is force-retired; one that still answers is left running and the call fails with `details.reason: 'daemon-drain-timeout'`, with a hint naming `agent-device daemon stop --state-dir <path>`.
79
- `@agent-device/testmu` is now a version 2 provider plugin. `import testmu from '@agent-device/testmu'` gives a factory, `testmu({ credentials: { username, accessKey } })`, that reads those credentials instead of `LT_USERNAME` and `LT_ACCESS_KEY`; without options it reads the variables as before. The package still declares the version 1 plugin entry, so `agent-device plugins add @agent-device/testmu` keeps working on hosts that predate the version 2 contract.

‎packages/daemon-contracts/src/daemon-registration-owner-startup-markers.test.ts‎

Lines changed: 111 additions & 152 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ import { appendFile, readFile, writeFile } from 'node:fs/promises';
22
import { spawnSync } from 'node:child_process';
33
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
44
import path from 'node:path';
5-
import { describe, expect, it } from 'vitest';
5+
import { expect, test } from 'vitest';
66
import {
77
claimDaemonStartupPhasesFile,
88
readDaemonStartupPhaseMarkers,
@@ -14,179 +14,138 @@ import {
1414
import { resolveDaemonPaths } from '@agent-device/daemon-contracts/daemon-resolution';
1515
import { mkdtempForTest } from '@agent-device/test-kit';
1616

17-
describe('per-launch startup marker files', () => {
18-
it('scopes marker files to the writing pid so a fresh launch never reads an older launch phases', async () => {
19-
const stateDir = await mkdtempForTest('agent-device-markers-');
20-
const first = resolveDaemonStartupPhasesPath(stateDir, 1111);
21-
const second = resolveDaemonStartupPhasesPath(stateDir, 2222);
22-
23-
recordDaemonStartupMarker(first, 'lock-acquired');
24-
25-
expect(second).not.toBe(first);
26-
expect(existsSync(second)).toBe(false);
27-
28-
recordDaemonStartupMarker(second, 'modules-loaded');
29-
const firstPhases = readDaemonStartupPhaseMarkers(readFileSync(first, 'utf8'));
30-
const secondPhases = readDaemonStartupPhaseMarkers(readFileSync(second, 'utf8'));
31-
expect(Object.keys(firstPhases)).toEqual(['lock-acquired']);
32-
expect(Object.keys(secondPhases)).toEqual(['modules-loaded']);
33-
});
17+
test('per-launch marker files: a fresh launch never reads an older launch phases', async () => {
18+
const stateDir = await mkdtempForTest('agent-device-markers-');
19+
const first = resolveDaemonStartupPhasesPath(stateDir, 1111);
20+
const second = resolveDaemonStartupPhasesPath(stateDir, 2222);
21+
recordDaemonStartupMarker(first, 'lock-acquired');
22+
expect(second).not.toBe(first);
23+
expect(existsSync(second)).toBe(false);
24+
recordDaemonStartupMarker(second, 'modules-loaded');
25+
const firstPhases = readDaemonStartupPhaseMarkers(readFileSync(first, 'utf8'));
26+
const secondPhases = readDaemonStartupPhaseMarkers(readFileSync(second, 'utf8'));
27+
expect(Object.keys(firstPhases)).toEqual(['lock-acquired']);
28+
expect(Object.keys(secondPhases)).toEqual(['modules-loaded']);
29+
});
3430

35-
it('keeps the earliest record when a replayed phase is appended after a later one', () => {
36-
const text = [
37-
'{"data":{"startupPhase":"listening","timestampMs":500}}',
38-
'{"data":{"startupPhase":"lock-acquired","timestampMs":20}}',
39-
'{"data":{"startupPhase":"lock-acquired","timestampMs":35}}',
40-
].join('\n');
41-
const phases = readDaemonStartupPhaseMarkers(text);
42-
expect(phases['lock-acquired']).toBe(20);
43-
expect(phases.listening).toBe(500);
44-
});
31+
test('marker reading keeps the earliest record when a phase replays after a later one', () => {
32+
const text = [
33+
'{"data":{"startupPhase":"listening","timestampMs":500}}',
34+
'{"data":{"startupPhase":"lock-acquired","timestampMs":20}}',
35+
'{"data":{"startupPhase":"lock-acquired","timestampMs":35}}',
36+
].join('\n');
37+
const phases = readDaemonStartupPhaseMarkers(text);
38+
expect(phases['lock-acquired']).toBe(20);
39+
expect(phases.listening).toBe(500);
40+
});
4541

46-
it('ignores unknown phases and unparsable lines so the protocol stays additive', () => {
47-
const text = [
48-
'{"data":{"startupPhase":"heartbeat","timestampMs":10}}',
49-
'{"data":{"startupPhase":"registered","timestampMs":"7"}}',
50-
'not json',
51-
'{"data":{"startupPhase":"listening","timestampMs":30}}',
52-
].join('\n');
53-
expect(readDaemonStartupPhaseMarkers(text)).toEqual({ listening: 30 });
54-
});
42+
test('marker reading ignores unknown phases and unparsable lines', () => {
43+
const text = [
44+
'{"data":{"startupPhase":"heartbeat","timestampMs":10}}',
45+
'{"data":{"startupPhase":"registered","timestampMs":"7"}}',
46+
'not json',
47+
'{"data":{"startupPhase":"listening","timestampMs":30}}',
48+
].join('\n');
49+
expect(readDaemonStartupPhaseMarkers(text)).toEqual({ listening: 30 });
5550
});
5651

57-
describe('producer lifetime claim', () => {
58-
it('empties a same-pid leftover so only the fresh lifetime markers survive', async () => {
59-
// The scenario the per-pid name alone cannot cover: both the client and the daemon of an old
60-
// launch died before either cleanup ran, the host reused the pid, and the new writer finds the
61-
// old lifetime's record under its own name. The entry claims its record before its first
62-
// marker; this child repeats that producer sequence against its own pid, which no test can
63-
// arrange from the outside.
64-
const stateDir = await mkdtempForTest('agent-device-markers-');
65-
writeFileSync(path.join(stateDir, 'package.json'), '{"type":"module"}');
66-
const entry = path.join(stateDir, 'claim-producer.js');
67-
const registrationUrl = new URL('./daemon-registration-owner.ts', import.meta.url).href;
68-
writeFileSync(
69-
entry,
70-
`import { claimDaemonStartupPhasesFile, recordDaemonStartupMarker, resolveDaemonStartupPhasesPath } from ${JSON.stringify(registrationUrl)};
52+
test('the producer lifetime claim empties a same-pid leftover', async () => {
53+
// Pid reuse after both the old client and daemon died before either cleanup: the entry claims
54+
// its record before its first marker; this child repeats that producer sequence itself.
55+
const stateDir = await mkdtempForTest('agent-device-markers-');
56+
writeFileSync(path.join(stateDir, 'package.json'), '{"type":"module"}');
57+
const entry = path.join(stateDir, 'claim-producer.js');
58+
const registrationUrl = new URL('./daemon-registration-owner.ts', import.meta.url).href;
59+
writeFileSync(
60+
entry,
61+
`import { claimDaemonStartupPhasesFile, recordDaemonStartupMarker, resolveDaemonStartupPhasesPath } from ${JSON.stringify(registrationUrl)};
7162
const file = resolveDaemonStartupPhasesPath(${JSON.stringify(stateDir)}, process.pid);
7263
recordDaemonStartupMarker(file, 'listening');
7364
recordDaemonStartupMarker(file, 'registered');
7465
claimDaemonStartupPhasesFile(file);
7566
recordDaemonStartupMarker(file, 'modules-loaded');
7667
`,
77-
);
78-
const child = spawnSync(process.execPath, ['--experimental-strip-types', entry], {
79-
encoding: 'utf8',
80-
});
81-
if (child.status !== 0) throw new Error(child.stderr);
82-
const phases = readDaemonStartupPhaseMarkers(
83-
readFileSync(resolveDaemonStartupPhasesPath(stateDir, child.pid!), 'utf8'),
84-
);
85-
expect(phases).toEqual({ 'modules-loaded': expect.any(Number) });
86-
});
87-
88-
it('reports a refused reset through the callback instead of throwing', async () => {
89-
const stateDir = await mkdtempForTest('agent-device-markers-');
90-
// A directory in place of the marker file makes the unlink fail.
91-
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 778);
92-
mkdirSync(markerPath, { recursive: true });
93-
94-
const reported: unknown[] = [];
95-
expect(() =>
96-
claimDaemonStartupPhasesFile(markerPath, (error) => {
97-
reported.push(error);
98-
}),
99-
).not.toThrow();
100-
101-
expect(reported).toHaveLength(1);
102-
expect(reported[0]).toHaveProperty('message');
68+
);
69+
const child = spawnSync(process.execPath, ['--experimental-strip-types', entry], {
70+
encoding: 'utf8',
10371
});
72+
if (child.status !== 0) throw new Error(child.stderr);
73+
const phases = readDaemonStartupPhaseMarkers(
74+
readFileSync(resolveDaemonStartupPhasesPath(stateDir, child.pid!), 'utf8'),
75+
);
76+
expect(phases).toEqual({ 'modules-loaded': expect.any(Number) });
10477
});
10578

106-
describe('registration-time sweep', () => {
107-
it('retires marker files of provably-dead pids and keeps files of live pids', async () => {
108-
const paths = resolveDaemonPaths(await mkdtempForTest('agent-device-markers-'));
109-
const deadPid = spawnDeadPid();
110-
const own = resolveDaemonStartupPhasesPath(paths.baseDir, process.pid);
111-
const dead = resolveDaemonStartupPhasesPath(paths.baseDir, deadPid);
112-
const legacy = path.join(paths.baseDir, 'daemon-startup-phases.jsonl');
113-
await Promise.all([
114-
writeFile(own, 'x\n', 'utf8'),
115-
writeFile(dead, 'x\n', 'utf8'),
116-
writeFile(legacy, 'x\n', 'utf8'),
117-
]);
118-
119-
const attempt = await tryAcquireDaemonRegistration(paths);
120-
expect(attempt.status).toBe('acquired');
121-
if (attempt.status !== 'acquired') throw new Error('registration refused');
122-
123-
expect(existsSync(dead)).toBe(false);
124-
expect(existsSync(own)).toBe(true);
125-
// The sweep retires only its own kind: an unknown entry stays as evidence.
126-
expect(existsSync(legacy)).toBe(true);
127-
await attempt.owner.finish();
128-
});
129-
130-
it('retires files whose pid is unparseable', async () => {
131-
const paths = resolveDaemonPaths(await mkdtempForTest('agent-device-markers-'));
132-
const stale = path.join(paths.baseDir, 'daemon-startup-phases-.jsonl');
133-
await writeFile(stale, 'x\n', 'utf8');
134-
135-
const attempt = await tryAcquireDaemonRegistration(paths);
136-
expect(attempt.status).toBe('acquired');
137-
if (attempt.status !== 'acquired') throw new Error('registration refused');
138-
139-
expect(existsSync(stale)).toBe(false);
140-
await attempt.owner.finish();
141-
});
79+
test('a refused claim reset reports through the callback instead of throwing', async () => {
80+
const stateDir = await mkdtempForTest('agent-device-markers-');
81+
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 778);
82+
mkdirSync(markerPath, { recursive: true });
83+
const reported: unknown[] = [];
84+
expect(() =>
85+
claimDaemonStartupPhasesFile(markerPath, (error) => {
86+
reported.push(error);
87+
}),
88+
).not.toThrow();
89+
expect(reported).toHaveLength(1);
90+
expect(reported[0]).toHaveProperty('message');
14291
});
14392

144-
describe('removeDaemonStartupPhasesFile', () => {
145-
it('removes a marker file and tolerates a file already gone', async () => {
146-
const stateDir = await mkdtempForTest('agent-device-markers-');
147-
const file = resolveDaemonStartupPhasesPath(stateDir, 4242);
148-
await writeFile(file, 'x\n', 'utf8');
149-
150-
removeDaemonStartupPhasesFile(file);
151-
expect(existsSync(file)).toBe(false);
152-
153-
expect(() => removeDaemonStartupPhasesFile(file)).not.toThrow();
154-
});
93+
test('acquiring the registration retires files of dead or unowned pids only', async () => {
94+
const paths = resolveDaemonPaths(await mkdtempForTest('agent-device-markers-'));
95+
const dead = resolveDaemonStartupPhasesPath(paths.baseDir, spawnDeadPid());
96+
const own = resolveDaemonStartupPhasesPath(paths.baseDir, process.pid);
97+
const unparseable = path.join(paths.baseDir, 'daemon-startup-phases-.jsonl');
98+
const legacy = path.join(paths.baseDir, 'daemon-startup-phases.jsonl');
99+
await Promise.all([
100+
writeFile(own, 'x\n', 'utf8'),
101+
writeFile(dead, 'x\n', 'utf8'),
102+
writeFile(unparseable, 'x\n', 'utf8'),
103+
writeFile(legacy, 'x\n', 'utf8'),
104+
]);
105+
const attempt = await tryAcquireDaemonRegistration(paths);
106+
expect(attempt.status).toBe('acquired');
107+
if (attempt.status !== 'acquired') throw new Error('registration refused');
108+
expect(existsSync(dead)).toBe(false);
109+
expect(existsSync(unparseable)).toBe(false);
110+
expect(existsSync(own)).toBe(true);
111+
// The sweep retires only its own kind: an unknown entry stays as evidence.
112+
expect(existsSync(legacy)).toBe(true);
113+
await attempt.owner.finish();
155114
});
156115

157-
describe('recordDaemonStartupMarker failure seam', () => {
158-
it('reports a write failure through the callback instead of throwing, keeping startup running', async () => {
159-
const stateDir = await mkdtempForTest('agent-device-markers-');
160-
// A directory in place of the marker file makes every append fail.
161-
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 777);
162-
mkdirSync(markerPath, { recursive: true });
163-
164-
const reported: unknown[] = [];
165-
expect(() =>
166-
recordDaemonStartupMarker(markerPath, 'modules-loaded', (error) => {
167-
reported.push(error);
168-
}),
169-
).not.toThrow();
170-
171-
expect(reported).toHaveLength(1);
172-
expect(reported[0]).toHaveProperty('message');
173-
});
174-
175-
it('keeps appending valid JSONL when an incomplete line precedes a crash-replay append', async () => {
176-
const stateDir = await mkdtempForTest('agent-device-markers-');
177-
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 999);
178-
await appendFile(markerPath, '{"data":{"startupPhase":"modules-lo', 'utf8');
116+
test('removeDaemonStartupPhasesFile removes a file and tolerates one already gone', async () => {
117+
const stateDir = await mkdtempForTest('agent-device-markers-');
118+
const file = resolveDaemonStartupPhasesPath(stateDir, 4242);
119+
await writeFile(file, 'x\n', 'utf8');
120+
removeDaemonStartupPhasesFile(file);
121+
expect(existsSync(file)).toBe(false);
122+
expect(() => removeDaemonStartupPhasesFile(file)).not.toThrow();
123+
});
179124

180-
recordDaemonStartupMarker(markerPath, 'composition-loaded');
125+
test('a marker write failure reports through the callback, keeping startup running', async () => {
126+
const stateDir = await mkdtempForTest('agent-device-markers-');
127+
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 777);
128+
mkdirSync(markerPath, { recursive: true });
129+
const reported: unknown[] = [];
130+
expect(() =>
131+
recordDaemonStartupMarker(markerPath, 'modules-loaded', (error) => {
132+
reported.push(error);
133+
}),
134+
).not.toThrow();
135+
expect(reported).toHaveLength(1);
136+
expect(reported[0]).toHaveProperty('message');
137+
});
181138

182-
const tail = await readFile(markerPath, 'utf8');
183-
expect(Object.keys(readDaemonStartupPhaseMarkers(tail))).toEqual(['composition-loaded']);
184-
});
139+
test('a crash-replay append keeps the file valid JSONL', async () => {
140+
const stateDir = await mkdtempForTest('agent-device-markers-');
141+
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 999);
142+
await appendFile(markerPath, '{"data":{"startupPhase":"modules-lo', 'utf8');
143+
recordDaemonStartupMarker(markerPath, 'composition-loaded');
144+
const tail = await readFile(markerPath, 'utf8');
145+
expect(Object.keys(readDaemonStartupPhaseMarkers(tail))).toEqual(['composition-loaded']);
185146
});
186147

187148
function spawnDeadPid(): number {
188-
// A process that has already exited proves the liveness check rejects it,
189-
// the same way the daemon-process takeover guards prove it.
190149
const child = spawnSync(process.execPath, ['-e', ''], { encoding: 'utf8' });
191150
if (child.error) throw child.error;
192151
const pid = child.pid;

0 commit comments

Comments
 (0)