Skip to content

Commit 28b9789

Browse files
thymikeeclaude
andauthored
fix(daemon-client): wait for an acknowledged quiesce instead of killing the daemon (#3780)
* fix(daemon-client): wait for an acknowledged quiesce to finish instead of killing the daemon After the daemon answers draining, ensureDaemon waits up to 60 s for it to exit and release its registration. Signals are sent only to a daemon still running at the deadline that also fails its liveness probes; one that still answers fails with daemon-drain-timeout. Part of #3747, refs #3754 Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * fix(daemon-client): report drain timeout for an embedded host and tighten drain tests Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * docs(daemon-client): say what an unverified retirement identity means Co-Authored-By: Claude Sonnet 5.5 <[email protected]> --------- Co-authored-by: Claude Sonnet 5.5 <[email protected]>
1 parent c1639b4 commit 28b9789

8 files changed

Lines changed: 417 additions & 14 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## Unreleased
44

5+
- Fix `ensureDaemon()`, the CLI and MCP killing a daemon they had just asked to leave. After the daemon answers the quiesce request with `draining`, the client now waits up to 60 s for it to exit by itself and clean up its registration, so plugin disposal and lease finalization finish (on Windows a signal ends the process outright, and on POSIX a drain longer than 3 s was killed). A daemon still running at the deadline that fails its liveness probes is force-retired; one that still answers is left running and the call fails with `details.reason: 'daemon-drain-timeout'`, with a hint naming `agent-device daemon stop --state-dir <path>`.
56
- Behavior change: the CLI and MCP now call `ensureDaemon()` explicitly once their config is resolved, then attach to the daemon it leaves. A command that finds a daemon from another version or build replaces it only if the daemon reports no request, session, repair session or live lease. A resumable upload still streaming and a host human-control hold that names no lease are not counted, so a replacement can end them. A daemon that answers nothing on any transport is retired after repeated failed liveness probes, which cannot show it idle. A busy one fails the command with `details.reason: 'daemon-busy'` instead of being replaced with its sessions, and a daemon from a release that predates quiescing fails it with `'daemon-incompatible'`; stop either with `agent-device daemon stop --state-dir <path>`. The takeover notice moves from `Replacing daemon … : <reason>` to `Replaced idle daemon …` (an incompatible daemon) or `Replaced unreachable daemon …` (one that answered nothing). A request that times out no longer retires an unresponsive daemon from the CLI or MCP; the error names `agent-device daemon stop --state-dir <path>`. A one-shot `replay` or `test` with an explicit `--state-dir` leaves the daemon it started running until its idle reap instead of stopping it. Remote `--daemon-base-url` connections and replay on its private ephemeral daemon are unchanged. Node.js clients keep `daemonLifecycle: 'managed'` until 0.22.
67
- Add `ensureDaemon({ stateDir })` and the client config `daemonLifecycle: 'attach' | 'managed'`. An `attach` client connects to the daemon registered in the state directory and never starts, replaces or kills one, including after a request times out. It fails with `details.reason: 'daemon-not-running'` (the hint points at `ensureDaemon()`) or `'daemon-incompatible'` (both identities in `details`, the hint names `agent-device daemon stop`). `ensureDaemon()` starts a missing daemon and replaces an incompatible one that answers only through an authenticated quiesce request the daemon answers itself (a resumable upload still streaming and a human-control hold that names no lease are not counted, so a replacement can end them): it refuses with `daemon-busy` while any request is in flight or any session, repair session or live lease exists, and a daemon from a release before this one is never replaced. A registered daemon that answers nothing is retired, as is one that times out on the quiesce request and then fails repeated liveness probes; neither can be asked. A remote `daemonBaseUrl` connection never starts or retires a local daemon in either mode. `managed` stays the default and behaves as before until 0.22.
78
- `lease_allocate` (and `client.leases.allocate()`) with a `leaseProvider` that no registered runtime claims now refuses before recording a lease, with `UNSUPPORTED_OPERATION`, `error.details.reason: provider-not-registered`, `details.provider` and `details.registeredProviders`. Previously a plugin provider that the daemon never loaded passed admission and recorded a lease with no device behind it. Leases with no provider (a `leaseBackend` alone included), `proxy` and `cloud` need no runtime and still allocate. Restart the daemon with the provider configured, then retry.

‎docs/adr/0006-daemon-rpc-protocol-version.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,8 +103,14 @@ refuses with `daemon-busy` and reopens admission in that same step. Otherwise it
103103
it joins neither the in-flight count nor the command registry.
104104

105105
The daemon accepts a quiesce only when it can prove its own start time and the request names both
106-
the pid and that start time; a daemon that cannot is never addressed. A client never signals a
107-
daemon that did not accept one, except to retire it after it answered `draining`.
106+
the pid and that start time; a daemon that cannot is never addressed.
107+
108+
After `draining` the client signals nothing and waits, up to a bounded deadline, for the daemon to
109+
exit by itself and release its registration: the daemon runs its own shutdown, which a signal would
110+
cut short. Only a daemon still running at the deadline that also fails its liveness probes is
111+
force-retired, and only when its command line proves it is a daemon process. A daemon that still
112+
answers is left alone and the call fails with `daemon-drain-timeout`. A client never signals a
113+
daemon that did not accept a quiesce, except an unreachable one it retires deliberately.
108114

109115
The census counts what the contract names and nothing else. A resumable upload still streaming and
110116
a host-administration human-control hold that names no lease are outside it, the same boundary
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
import assert from 'node:assert/strict';
2+
import { afterEach, test, vi } from 'vitest';
3+
import { closeLoopbackServer, supportsLoopbackBind } from '@agent-device/test-kit';
4+
import { AppError } from '@agent-device/kernel/errors';
5+
import { isProcessAlive } from '@agent-device/host-kit/process';
6+
import { finishRegisteredDaemonFixtures } from '../../__tests__/test-utils/registered-daemon-fixture.ts';
7+
import {
8+
draining,
9+
launchesFreshDaemonAt,
10+
okResult,
11+
registerDaemon,
12+
startScriptedDaemon,
13+
stateDirAndPaths,
14+
} from './explicit-lifecycle-fixture.ts';
15+
16+
vi.mock('@agent-device/host-kit/command', async (importOriginal) => ({
17+
...(await importOriginal<typeof import('@agent-device/host-kit/command')>()),
18+
runCmdDetachedMonitored: vi.fn(),
19+
}));
20+
// An embedded host registered by some other process runs a command that is no daemon entry.
21+
let hostedByAnotherProcess = false;
22+
vi.mock('@agent-device/host-kit/process', async (importOriginal) => {
23+
const actual = await importOriginal<typeof import('@agent-device/host-kit/process')>();
24+
return {
25+
...actual,
26+
readProcessCommand: (pid: number) =>
27+
hostedByAnotherProcess ? 'node /opt/app/server.js' : actual.readProcessCommand(pid),
28+
};
29+
});
30+
vi.mock('@agent-device/host-kit/retry', async (importOriginal) => ({
31+
...(await importOriginal<typeof import('@agent-device/host-kit/retry')>()),
32+
sleep: vi.fn(),
33+
}));
34+
import { runCmdDetachedMonitored } from '@agent-device/host-kit/command';
35+
import { sleep } from '@agent-device/host-kit/retry';
36+
import { ensureDaemon } from '../daemon-client-ensure.ts';
37+
38+
// A daemon that accepted a quiesce request is leaving by itself. These tests give the drain a
39+
// virtual clock that starts when the daemon says `draining`, so a drain of any length costs no real
40+
// time, and watch which signals the client sends while it waits.
41+
42+
const spawn = vi.mocked(runCmdDetachedMonitored);
43+
const actualRetry = await vi.importActual<typeof import('@agent-device/host-kit/retry')>(
44+
'@agent-device/host-kit/retry',
45+
);
46+
const realNow = Date.now.bind(Date);
47+
const actualKill = process.kill.bind(process);
48+
49+
// What the old retirement waited before it escalated to SIGKILL; a healthy drain may outlast it.
50+
const OLD_GRACE_MS = 3_000;
51+
// `DRAIN_DEADLINE_MS` in `daemon-client-ensure.ts`.
52+
const DRAIN_DEADLINE_MS = 60_000;
53+
54+
afterEach(async () => {
55+
hostedByAnotherProcess = false;
56+
vi.restoreAllMocks();
57+
spawn.mockReset();
58+
vi.mocked(sleep).mockReset();
59+
await finishRegisteredDaemonFixtures();
60+
});
61+
62+
type VirtualDrain = {
63+
/** Virtual milliseconds since the daemon answered `draining`. */
64+
elapsedMs: () => number;
65+
/** Starts the clock; called by the stand-in daemon as it answers. */
66+
begin: (leavesAfterMs: number | undefined) => void;
67+
/** Ends the virtual phase and returns to the real clock. */
68+
end: () => void;
69+
signals: Array<{ signal: NodeJS.Signals | number; atMs: number }>;
70+
};
71+
72+
/**
73+
* Time only moves while the client waits for the drain, and signals to \`pid\` are recorded with the
74+
* virtual time they arrived at. \`leavesAfterMs\` is when the daemon finishes leaving by itself.
75+
*/
76+
function installVirtualDrain(pid: () => number): VirtualDrain {
77+
let virtual = false;
78+
let now = 0;
79+
let startedAt = 0;
80+
let leavesAt: number | undefined;
81+
const signals: VirtualDrain['signals'] = [];
82+
vi.spyOn(Date, 'now').mockImplementation(() => (virtual ? now : realNow()));
83+
vi.mocked(sleep).mockImplementation(async (ms: number) => {
84+
if (!virtual) return await actualRetry.sleep(ms);
85+
now += ms;
86+
if (leavesAt !== undefined && now >= leavesAt) {
87+
leavesAt = undefined;
88+
actualKill(pid(), 'SIGTERM');
89+
await waitForExit(pid());
90+
virtual = false;
91+
}
92+
});
93+
vi.spyOn(process, 'kill').mockImplementation(((
94+
target: number,
95+
signal?: NodeJS.Signals | number,
96+
) => {
97+
if (target === pid() && signal !== undefined && signal !== 0) {
98+
signals.push({ signal, atMs: now - startedAt });
99+
virtual = false;
100+
}
101+
return actualKill(target, signal ?? 'SIGTERM');
102+
}) as typeof process.kill);
103+
return {
104+
elapsedMs: () => now - startedAt,
105+
begin: (leavesAfterMs) => {
106+
now = realNow();
107+
startedAt = now;
108+
leavesAt = leavesAfterMs === undefined ? undefined : now + leavesAfterMs;
109+
virtual = true;
110+
},
111+
end: () => {
112+
virtual = false;
113+
},
114+
signals,
115+
};
116+
}
117+
118+
async function waitForExit(pid: number): Promise<void> {
119+
for (let polls = 0; isProcessAlive(pid) && polls < 500; polls += 1) {
120+
await actualRetry.sleep(10);
121+
}
122+
}
123+
124+
async function withLoopback(t: { skip: (note: string) => void }): Promise<boolean> {
125+
if (await supportsLoopbackBind()) return true;
126+
t.skip('loopback listeners are not permitted in this environment');
127+
return false;
128+
}
129+
130+
test('a drain that outlasts the old kill window is waited out, not signaled', async (t) => {
131+
if (!(await withLoopback(t))) return;
132+
const { stateDir, paths } = stateDirAndPaths();
133+
let childPid = 0;
134+
const drain = installVirtualDrain(() => childPid);
135+
const stale = await startScriptedDaemon(draining, { afterRpc: () => drain.begin(5_000) });
136+
const fresh = await startScriptedDaemon(okResult({}));
137+
const { child } = await registerDaemon(paths, {
138+
httpPort: stale.port,
139+
version: '0.0.0-stale',
140+
capabilities: ['quiesce'],
141+
});
142+
childPid = child.pid;
143+
launchesFreshDaemonAt(spawn, paths, fresh.port);
144+
145+
try {
146+
const connection = await ensureDaemon({ stateDir, daemonTransport: 'http' });
147+
148+
assert.equal(connection.started, true);
149+
assert.ok(drain.elapsedMs() >= 5_000, 'the drain ran for 5 virtual seconds');
150+
assert.ok(drain.elapsedMs() > OLD_GRACE_MS);
151+
assert.deepEqual(drain.signals, [], 'the client signaled nothing while the daemon drained');
152+
assert.equal(isProcessAlive(childPid), false, 'the daemon left by itself');
153+
assert.equal(spawn.mock.calls.length, 1, 'its successor started after it left');
154+
} finally {
155+
await closeLoopbackServer(stale.server);
156+
await closeLoopbackServer(fresh.server);
157+
}
158+
});
159+
160+
test('a daemon still running past the deadline that answers no probe is force-retired', async (t) => {
161+
if (!(await withLoopback(t))) return;
162+
const { stateDir, paths } = stateDirAndPaths();
163+
let childPid = 0;
164+
const drain = installVirtualDrain(() => childPid);
165+
// A draining daemon has closed its servers, so it answers no liveness probe.
166+
const stale = await startScriptedDaemon(draining, {
167+
afterRpc: () => {
168+
drain.begin(undefined);
169+
void closeLoopbackServer(stale.server);
170+
},
171+
});
172+
const fresh = await startScriptedDaemon(okResult({}));
173+
const { child } = await registerDaemon(paths, {
174+
httpPort: stale.port,
175+
version: '0.0.0-stale',
176+
capabilities: ['quiesce'],
177+
});
178+
childPid = child.pid;
179+
launchesFreshDaemonAt(spawn, paths, fresh.port);
180+
181+
try {
182+
const connection = await ensureDaemon({ stateDir, daemonTransport: 'http' });
183+
184+
assert.equal(connection.started, true);
185+
assert.equal(drain.signals.length > 0, true, 'the stuck daemon was signaled');
186+
assert.ok(
187+
drain.signals.every(({ atMs }) => atMs >= DRAIN_DEADLINE_MS),
188+
`no signal before the deadline: ${JSON.stringify(drain.signals)}`,
189+
);
190+
assert.equal(isProcessAlive(childPid), false);
191+
assert.equal(spawn.mock.calls.length, 1);
192+
} finally {
193+
await closeLoopbackServer(fresh.server);
194+
}
195+
});
196+
197+
test('a daemon still running past the deadline that answers a probe is left alone', async (t) => {
198+
if (!(await withLoopback(t))) return;
199+
const { stateDir, paths } = stateDirAndPaths();
200+
let childPid = 0;
201+
const drain = installVirtualDrain(() => childPid);
202+
const stale = await startScriptedDaemon(draining, { afterRpc: () => drain.begin(undefined) });
203+
const { child } = await registerDaemon(paths, {
204+
httpPort: stale.port,
205+
version: '0.0.0-stale',
206+
capabilities: ['quiesce'],
207+
});
208+
childPid = child.pid;
209+
210+
try {
211+
await assert.rejects(ensureDaemon({ stateDir, daemonTransport: 'http' }), (error: unknown) => {
212+
assert.ok(error instanceof AppError);
213+
assert.equal(error.details?.reason, 'daemon-drain-timeout');
214+
assert.match(String(error.details?.hint), /agent-device daemon stop/);
215+
return true;
216+
});
217+
assert.deepEqual(drain.signals, []);
218+
assert.equal(isProcessAlive(childPid), true);
219+
assert.equal(spawn.mock.calls.length, 0);
220+
drain.end();
221+
} finally {
222+
await closeLoopbackServer(stale.server);
223+
}
224+
});
225+
226+
test('an embedded host that outlives the deadline and answers no probe is never signaled', async (t) => {
227+
if (!(await withLoopback(t))) return;
228+
const { stateDir, paths } = stateDirAndPaths();
229+
let childPid = 0;
230+
const drain = installVirtualDrain(() => childPid);
231+
const stale = await startScriptedDaemon(draining, {
232+
afterRpc: () => {
233+
drain.begin(undefined);
234+
void closeLoopbackServer(stale.server);
235+
},
236+
});
237+
const { child } = await registerDaemon(paths, {
238+
httpPort: stale.port,
239+
version: '0.0.0-stale',
240+
capabilities: ['quiesce'],
241+
});
242+
childPid = child.pid;
243+
hostedByAnotherProcess = true;
244+
245+
await assert.rejects(ensureDaemon({ stateDir, daemonTransport: 'http' }), (error: unknown) => {
246+
assert.ok(error instanceof AppError);
247+
assert.equal(error.details?.reason, 'daemon-drain-timeout');
248+
return true;
249+
});
250+
251+
assert.deepEqual(drain.signals, []);
252+
assert.equal(isProcessAlive(childPid), true);
253+
assert.equal(spawn.mock.calls.length, 0);
254+
drain.end();
255+
});

‎src/daemon-client/__tests__/daemon-client-ensuring.test.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import { isProcessAlive } from '@agent-device/host-kit/process';
66
import { finishRegisteredDaemonFixtures } from '../../__tests__/test-utils/registered-daemon-fixture.ts';
77
import {
88
busy,
9+
createDaemonLeaver,
910
draining,
1011
launchesFreshDaemonAt,
1112
okResult,
@@ -32,6 +33,7 @@ const spawn = vi.mocked(runCmdDetachedMonitored);
3233

3334
afterEach(async () => {
3435
vi.unstubAllEnvs();
36+
vi.restoreAllMocks();
3537
spawn.mockReset();
3638
await finishRegisteredDaemonFixtures();
3739
});
@@ -78,13 +80,15 @@ test('the first request ensures a missing daemon once and every request attaches
7880
test('an idle incompatible daemon is replaced, and the replacement is reported', async (t) => {
7981
if (!(await supportsLoopbackBind())) return t.skip('loopback listeners are not permitted');
8082
const { stateDir, paths } = stateDirAndPaths();
81-
const stale = await startScriptedDaemon(draining);
83+
const leaver = createDaemonLeaver();
84+
const stale = await startScriptedDaemon(draining, { afterRpc: leaver.leave });
8285
const fresh = await startScriptedDaemon(okResult({}));
8386
const { child } = await registerDaemon(paths, {
8487
httpPort: stale.port,
8588
version: '0.0.0-stale',
8689
capabilities: ['quiesce'],
8790
});
91+
leaver.pid = child.pid;
8892
launchesFreshDaemonAt(spawn, paths, fresh.port);
8993
const { sent, transport } = recordingTransport();
9094
const replaced: Array<{ stateDir: string; pid: number; reason: string }> = [];
@@ -100,6 +104,7 @@ test('an idle incompatible daemon is replaced, and the replacement is reported',
100104
{ stateDir: paths.baseDir, pid: child.pid, reason: 'incompatible' },
101105
]);
102106
assert.equal(sent[0]?.lifecycle, 'attach');
107+
assert.deepEqual(leaver.clientSignals, [], 'the client signaled nothing');
103108
} finally {
104109
await closeLoopbackServer(stale.server);
105110
await closeLoopbackServer(fresh.server);

0 commit comments

Comments
 (0)