Skip to content

Commit 5e48486

Browse files
authored
test: catch daemon lifecycle and durable-state leaks (#1781 B1) (#1859)
* test: daemon leak oracle around the real-subprocess daemon lanes (#1781 B1) Adds test/integration/support/daemon-leak-oracle.ts and calls it at the end of smoke-daemon-clean, smoke-daemon-http and daemon-replace-exit-flush. After shutdown the oracle asserts that no daemon-owned process survives (ownership: PPID descendant, PGID = daemon pid, AGENT_DEVICE_STATE_DIR env, state-dir argv — never global counts) and that the isolated state dir holds only classified artifacts (no *.tmp, no daemon.json/lock without a live daemon, no open capture descriptor). Red-proofs: pre-fix #1324 (a84caa8) leaves simctl recordVideo with ppid 1 / pgid = dead daemon; pre-fix #1109 (be4bd09) leaves the agent-browser daemon plus its Chrome fleet. Both are clean on main. Refs #1781 #1431 * test: fail the daemon leak oracle on a surviving daemon and pin its rules Review of #1859 found the oracle reported "clean" when the daemon itself outlived shutdown: daemon pids were excluded from ownership, liveDaemonPids never reached hasDaemonLeaks, and a live daemon even flipped daemon.json/lock from stray to expected. stopProcessForTakeover is best-effort void, so only smoke-daemon-clean independently asserted death. - a live daemon pid at phase 'after-shutdown' is now itself a leak, and its metadata files stay stray; a live daemon remains legitimate at 'after-close' - split the pure ownership/residue rules into daemon-leak-model.ts and pin them with daemon-leak-model.test.ts, using the real ps rows captured during the #1109 and #1324 red-proofs (the lanes' daemons own no children, so the fixture test is what guards those shapes in CI) - exempt the managed tools/ install tree before the .tmp rule, so agent-browser's own download temporaries are no longer a false LEAK - report empty directories as residue (an unswept session scaffold leaves no file) - reuse src/utils/host-process.ts (expandProcessTree, uniquePositivePids) and its /bin/ps convention instead of re-deriving them - assert in smoke-daemon-http on the success path, not in finally, so the settle window cannot replace a primary assertion's diagnostic Refs #1781 #1431 * test: make the leak oracle's after-close phase name the session that closed Re-review of #1859: `after-close` accepted every capture descriptor and legacy marker, so the closed session's unfinalized handle was indistinguishable from another session's legitimately live one — the phase could not fail, the same shape as the surviving-daemon blocker, and it left half of B1's stated scope undelivered. - the observation carries the session directories closed at the checkpoint; a capture handle must be finalized once its owning session is gone (every session after shutdown, only the closed ones after close), while another session's live handle stays expected and a legacy pid marker is never a finish record - the oracle accepts `closedSessions` and a `sessionsDir` override, normalizing entries to the canonical `sessions/<name>/…` shape so an in-process harness rooted directly at its own sessions dir is classified the same way - add a real route regression: a provider-backed session with a live screen recording is closed through the daemon route, and the oracle refuses a descriptor left `lifecycle: "open"`. Reverting the close-route finalization (session-close-lifecycle-teardown.ts, the #1325 fix) turns it red naming sessions/default/screen-recording.resource.json; with the pre-fix model it stays green, which is the P1 in one line Refs #1781 #1431 * test: require the closed-session identity at the leak oracle's owning interface Re-review of #1859: `closedSessions` was optional and defaulted to empty, so any caller — or the standalone CLI invoked with `--phase after-close` and no `--closed-session` — silently restored the vacuous checkpoint that accepts every unfinalized capture handle and reports clean. - phase and the identity that phase needs are now one discriminated shape: { phase: 'after-close'; closedSessions: NonEmpty<string> } | { phase: 'after-shutdown' }, so the empty case is not expressible and 'after-shutdown' is unchanged - the CLI refuses the same invocation with a typed INVALID_ARGS error and a hint naming what the missing identity would have cost, instead of degrading - daemon-leak-oracle-cli.test.ts drives the real CLI: the unnamed after-close invocation must fail without reporting, the named one finds the planted unfinalized handle, and after-shutdown is unaffected. Reverting the guard makes it print 'clean (after-close)' over that same handle and turns the test red Refs #1781 #1431 * test: split the leak oracle's unrecorded process arm out of the lane path Maintainer asked whether the harness can be trimmed. The two arms differ in kind: the daemon already records captures and state-dir artifacts, so those rules are short and fire on every lane run; nothing records what a daemon spawned, so the process arm reconstructs ownership from the OS four ways — and the three CLI daemon lanes are device-free, so it only ever compared an empty set to an empty set. Move that arm to test/integration/support/daemon-owned-process-probe.ts, the manual script that produced the #1109/#1324 evidence, and keep its rules pinned by a fixture test (no lane can run the probe, so its fixtures are the only CI guard on those shapes). The shipped oracle keeps every guarantee it had: surviving daemon at after-shutdown, the closed-session finalization rule, the state-dir allowlist, the CLI refusal, and the phase-discriminated types. Lane-path harness 852 → 548 LOC (-36%); repo total roughly flat, because the ownership reconstruction can be relocated but not deleted. That is the argument for the follow-up: once the daemon records owned child pids the way it records captures, the arm collapses to "read the record, assert they are dead" and moves back into the oracle. Re-proved after the move: #1324 against pre-fix a84caa8 on an iPhone 16 simulator still goes red through the probe (simctl recordVideo, ppid 1, pgid = the dead daemon, 0-byte mp4), and clean once the orphan is reaped. Refs #1781 #1431 * test: guard the daemon-owned-process arm in CI on the live web lane Re-review of ce9f0ee: moving that arm to a manual probe left the two failures B1 exists to prevent detectable only by hand, and a fixture test proves regexes recognize synthetic rows, not that the shipped route reaps what it spawned. Restore the arm to the shipped oracle (the three files return byte-identical to 3a5b9be) and give it a lane that can execute it: smoke-web-platform is the one CI route whose daemon owns real children — the managed agent-browser daemon and its Chrome fleet. After the normal smoke it reopens a session, stops the daemon with that session still open (the #1109 shape: an ordinary close reaps the fleet, so only an unclosed session can strand it), and requires that nothing owned outlives the browser idle window. Proven both ways locally: green in 59s, and red when the fleet is stranded (browser idle window raised past the settle budget) with the oracle naming 15 owned processes — the #1109 signature, in a lane that runs on every PR. Also: daemon-replace-exit-flush cleared `info` before the oracle ran, so a failed stop would skip the `finally` retry and remove the state dir while the daemon was still alive. Clear it only once the checkpoint passes. Refs #1781 #1431 #1882 * docs: align daemon leak coverage rationale * refactor(test): narrow daemon oracle to durable state leaks
1 parent b0f9c05 commit 5e48486

9 files changed

Lines changed: 741 additions & 15 deletions
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
import test from 'node:test';
2+
import assert from 'node:assert/strict';
3+
import fs from 'node:fs';
4+
import os from 'node:os';
5+
import path from 'node:path';
6+
import { runCmdSync } from '../../src/utils/exec.ts';
7+
8+
// #1781 B1: the oracle's `after-close` checkpoint is only meaningful when it can
9+
// name the sessions that closed — without them it would accept every unfinalized
10+
// capture handle and report clean, which is the vacuous mode the type system now
11+
// refuses in code. The standalone CLI is the one caller that builds options from
12+
// strings rather than types, so it must refuse the same invocation at runtime
13+
// instead of silently degrading to that mode.
14+
15+
const ORACLE_PATH = 'test/integration/support/daemon-leak-oracle.ts';
16+
17+
function runOracle(args: string[]) {
18+
return runCmdSync(process.execPath, ['--experimental-strip-types', ORACLE_PATH, ...args], {
19+
allowFailure: true,
20+
timeoutMs: 60_000,
21+
});
22+
}
23+
24+
test('the leak oracle CLI refuses an after-close checkpoint that names no session', (t) => {
25+
const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-leak-oracle-cli-'));
26+
// A closed session that left an unfinalized capture handle: the exact residue
27+
// the refused invocation would have reported clean.
28+
const sessionDir = path.join(stateDir, 'sessions', 'closed-one');
29+
fs.mkdirSync(sessionDir, { recursive: true });
30+
fs.writeFileSync(
31+
path.join(sessionDir, 'screen-recording.resource.json'),
32+
`${JSON.stringify({ lifecycle: 'open' })}\n`,
33+
);
34+
t.after(() => fs.rmSync(stateDir, { recursive: true, force: true }));
35+
36+
const refused = runOracle(['--state-dir', stateDir, '--phase', 'after-close']);
37+
assert.notEqual(refused.exitCode, 0, `expected a refusal, got:\n${refused.stdout}`);
38+
assert.match(refused.stderr, /--phase after-close requires at least one --closed-session/);
39+
// A refusal, not a leak report: the checkpoint never ran.
40+
assert.doesNotMatch(refused.stdout, /daemon leak oracle:/);
41+
42+
// The same invocation, once it names the session, runs and finds the handle.
43+
const named = runOracle([
44+
'--state-dir',
45+
stateDir,
46+
'--phase',
47+
'after-close',
48+
'--closed-session',
49+
'closed-one',
50+
'--settle-ms',
51+
'0',
52+
]);
53+
assert.equal(named.exitCode, 1, `expected a leak report, got:\n${named.stdout}${named.stderr}`);
54+
assert.match(named.stdout, /LEAK \(after-close\)/);
55+
assert.match(named.stdout, /sessions\/closed-one\/screen-recording\.resource\.json/);
56+
57+
// `after-shutdown` needs no session identity and is unaffected by the guard.
58+
const shutdown = runOracle(['--state-dir', stateDir, '--settle-ms', '0']);
59+
assert.equal(shutdown.exitCode, 1, shutdown.stderr);
60+
assert.match(shutdown.stdout, /LEAK \(after-shutdown\)/);
61+
});

‎test/integration/daemon-replace-exit-flush.test.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import path from 'node:path';
66
import { skipWhenLoopbackUnavailable } from '../../src/__tests__/test-utils/loopback.ts';
77
import { stopProcessForTakeover } from '../../src/daemon/daemon-process.ts';
88
import { isProcessAlive } from '../../src/utils/host-process.ts';
9+
import { assertNoDaemonLeaks } from './support/daemon-leak-oracle.ts';
910
import { runCliJson } from './test-helpers.ts';
1011

1112
// #1596: a CLI command that finds its recorded daemon unreachable replaces it
@@ -28,6 +29,7 @@ test('daemon replace mid-command returns a structured, parseable error and exits
2829

2930
const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replace-exit-flush-'));
3031
let info: DaemonInfo | null = null;
32+
const daemonPids: number[] = [];
3133
try {
3234
// A real daemon, started by this codebase, so its recorded version/code
3335
// signature legitimately match — the only way to reach the "unreachable"
@@ -36,6 +38,7 @@ test('daemon replace mid-command returns a structured, parseable error and exits
3638
assert.equal(started.status, 0, `${started.stderr}\n${started.stdout}`);
3739

3840
info = readDaemonInfo(stateDir);
41+
daemonPids.push(info.pid);
3942
assert.equal(isProcessAlive(info.pid), true, 'expected the started daemon to be alive');
4043

4144
// Kill it out from under its own metadata: daemon.json stays put and
@@ -69,6 +72,19 @@ test('daemon replace mid-command returns a structured, parseable error and exits
6972
);
7073

7174
info = readDaemonInfo(stateDir);
75+
daemonPids.push(info.pid);
76+
await stopProcessForTakeover(info.pid, {
77+
termTimeoutMs: 1_500,
78+
killTimeoutMs: 1_500,
79+
expectedStartTime: info.processStartTime,
80+
});
81+
// #1781 B1: neither the SIGKILLed daemon nor its replacement may leave owned
82+
// processes or unclassified state-dir residue once both are gone. `info`
83+
// stays set until this passes: `stopProcessForTakeover` is best-effort, so a
84+
// failed stop must still reach the `finally` retry below rather than have
85+
// the state dir removed out from under a daemon that is still running.
86+
await assertNoDaemonLeaks({ stateDir, daemonPids, phase: 'after-shutdown' });
87+
info = null;
7288
} finally {
7389
if (info) {
7490
await stopProcessForTakeover(info.pid, {
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
import assert from 'node:assert/strict';
2+
import path from 'node:path';
3+
import { expect, test } from 'vitest';
4+
import { assertRpcOk } from './assertions.ts';
5+
import {
6+
createAndroidRecordingScenarioHarness,
7+
withAndroidRecordingScenario,
8+
} from './android-recording-fixtures.ts';
9+
import { createAndroidRecordingProvider } from './android-recording-provider-fixtures.ts';
10+
import { PROVIDER_SCENARIO_ANDROID } from './fixtures.ts';
11+
import { PARALLEL_PROVIDER_SCENARIO_TIMEOUT_MS } from './test-timeouts.ts';
12+
import { screenRecordingResourceStore } from '../../../src/daemon/screen-recording-resource-store.ts';
13+
import { assertNoDaemonLeaks } from '../support/daemon-leak-oracle.ts';
14+
15+
// The `after-close` half of the #1781 B1 leak oracle, on a real daemon route.
16+
// The three CLI daemon lanes only ever shut a daemon down, so this is the one
17+
// place a closing session actually owns a resource: `record start` opens a
18+
// record-only session with a live screen-recording handle, and the session-close
19+
// teardown that #1325 added must finalize it. Removing that teardown step leaves
20+
// the descriptor `lifecycle: "open"` and turns this test red.
21+
test(
22+
'Provider-backed integration closing a recording session leaves no unfinalized capture handle',
23+
async () => {
24+
await withAndroidRecordingScenario(
25+
'agent-device-provider-scenario-close-leak-',
26+
async (tmpDir) => {
27+
const calls: string[][] = [];
28+
const outputPath = path.join(tmpDir, 'close-leak.mp4');
29+
const daemon = await createAndroidRecordingScenarioHarness({
30+
androidAdbProvider: () => createAndroidRecordingProvider({ calls }),
31+
deviceInventoryProvider: async () => [PROVIDER_SCENARIO_ANDROID],
32+
});
33+
try {
34+
const started = await daemon.callCommand('record', ['start', outputPath], {
35+
platform: 'android',
36+
serial: PROVIDER_SCENARIO_ANDROID.id,
37+
recordingScope: 'device',
38+
});
39+
assert.equal(assertRpcOk<{ recording?: unknown }>(started).recording, 'started');
40+
41+
const sessionDir = daemon.sessionDir('default');
42+
// The live handle is durable state: the oracle must be able to see the
43+
// unfinalized descriptor it will later refuse.
44+
assert.equal(
45+
screenRecordingResourceStore.read(screenRecordingResourceStore.resolvePath(sessionDir))
46+
.status,
47+
'decoded',
48+
);
49+
50+
await daemon.callCommand('close', [], {
51+
platform: 'android',
52+
serial: PROVIDER_SCENARIO_ANDROID.id,
53+
});
54+
55+
// `daemonPids: []` on purpose: this harness runs the daemon route
56+
// in-process, so there is no daemon process to own children and the
57+
// state-dir arm is the whole check here. The process arm is exercised by
58+
// the CLI lanes and pinned by daemon-leak-model.test.ts.
59+
await expect(
60+
assertNoDaemonLeaks({
61+
stateDir: path.dirname(sessionDir),
62+
sessionsDir: path.dirname(sessionDir),
63+
daemonPids: [],
64+
phase: 'after-close',
65+
closedSessions: [path.basename(sessionDir)],
66+
settleMs: 0,
67+
}),
68+
).resolves.toBeUndefined();
69+
} finally {
70+
await daemon.close();
71+
}
72+
},
73+
);
74+
},
75+
PARALLEL_PROVIDER_SCENARIO_TIMEOUT_MS,
76+
);

‎test/integration/smoke-daemon-clean.test.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import { skipWhenLoopbackUnavailable } from '../../src/__tests__/test-utils/loop
77
import { runCmdSync } from '../../src/utils/exec.ts';
88
import { stopProcessForTakeover } from '../../src/daemon/daemon-process.ts';
99
import { isProcessAlive } from '../../src/utils/host-process.ts';
10+
import { assertNoDaemonLeaks } from './support/daemon-leak-oracle.ts';
1011
import { runCliJson } from './test-helpers.ts';
1112

1213
type DaemonInfo = {
@@ -41,6 +42,9 @@ test('clean daemon script stops a live daemon before removing metadata', async (
4142
assert.equal(isProcessAlive(info.pid), false);
4243
assert.equal(fs.existsSync(path.join(stateDir, 'daemon.json')), false);
4344
assert.equal(fs.existsSync(path.join(stateDir, 'daemon.lock')), false);
45+
// #1781 B1: the stopped daemon must take every process it owned with it and
46+
// leave only classified artifacts in its state dir.
47+
await assertNoDaemonLeaks({ stateDir, daemonPids: [info.pid], phase: 'after-shutdown' });
4448
} finally {
4549
if (info) {
4650
await stopProcessForTakeover(info.pid, {

‎test/integration/smoke-daemon-http.test.ts‎

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import { DAEMON_RPC_PROTOCOL_VERSION } from '../../src/daemon/http-health.ts';
77
import { skipWhenLoopbackUnavailable } from '../../src/__tests__/test-utils/loopback.ts';
88
import { stopProcessForTakeover } from '../../src/daemon/daemon-process.ts';
99
import { formatResultDebug } from './cli-json.ts';
10+
import { assertNoDaemonLeaks } from './support/daemon-leak-oracle.ts';
1011
import { runCliJson } from './test-helpers.ts';
1112

1213
type DaemonInfo = {
@@ -65,8 +66,17 @@ test('daemon HTTP transport starts from CLI and accepts a command RPC', async (t
6566
const unauthorized = await callCommandRpc({ ...info, token: 'wrong-token' }, 'session_list');
6667
assert.equal(unauthorized.status, 401);
6768
assert.equal(unauthorized.body.error?.data?.code, 'UNAUTHORIZED');
69+
// #1781 B1: the HTTP-mode daemon must exit — leaving nothing it owns and no
70+
// unclassified state-dir residue. Asserted on the success path so the
71+
// oracle's settle window can never replace a primary assertion's
72+
// diagnostic; the `finally` below stays best-effort cleanup.
73+
await stopDaemon(info);
74+
await assertNoDaemonLeaks({ stateDir, daemonPids: [info.pid], phase: 'after-shutdown' });
6875
} finally {
69-
await stopDaemonForStateDir(stateDir);
76+
if (fs.existsSync(path.join(stateDir, 'daemon.json'))) {
77+
await stopDaemon(readDaemonInfo(stateDir));
78+
}
79+
fs.rmSync(stateDir, { recursive: true, force: true });
7080
}
7181
});
7282

@@ -101,18 +111,11 @@ async function callCommandRpc(
101111
};
102112
}
103113

104-
async function stopDaemonForStateDir(stateDir: string): Promise<void> {
105-
try {
106-
const infoPath = path.join(stateDir, 'daemon.json');
107-
if (!fs.existsSync(infoPath)) return;
108-
const info = readDaemonInfo(stateDir);
109-
if (!Number.isInteger(info.pid) || info.pid <= 0) return;
110-
await stopProcessForTakeover(info.pid, {
111-
termTimeoutMs: 1500,
112-
killTimeoutMs: 1500,
113-
expectedStartTime: info.processStartTime,
114-
});
115-
} finally {
116-
fs.rmSync(stateDir, { recursive: true, force: true });
117-
}
114+
async function stopDaemon(info: DaemonInfo): Promise<void> {
115+
if (!Number.isInteger(info.pid) || info.pid <= 0) return;
116+
await stopProcessForTakeover(info.pid, {
117+
termTimeoutMs: 1500,
118+
killTimeoutMs: 1500,
119+
expectedStartTime: info.processStartTime,
120+
});
118121
}

0 commit comments

Comments
 (0)