Skip to content

Commit e4c3b42

Browse files
authored
test: refuse foreign-pid signals from unit-test workers (Coverage fork death, #1824) (#1854)
* ci: w3-1824 experiment — trace fork signals and plant pid sentinels in the Coverage job Temporary instrumentation for #1824. Every vitest fork logs each real process.kill it sends to a foreign pid (and every kill/pkill it spawns); the Coverage job parks sentinel processes on the pids the Apple runner tests fabricate (4141/4242/4343/4444) and reports which of them survive the run. Reverted before this PR leaves draft. * test: refuse foreign-pid signals from unit-test workers A vitest worker may signal only itself and the processes it spawned. src/__tests__/hermetic-signal-setup.ts records any other process.kill, answers it with ESRCH (so best-effort kill paths proceed as if the pid were dead), and fails the sending test by name in afterEach. The senders this catches today are the Apple runner tests, which fabricate runner child pids (4242, 4141, 4343, 4444) and mocked the liveness reads in host-process.ts but not the signal writes: killRunnerProcessTree delivered real SIGINT/SIGTERM/SIGKILL to those pids and their process groups — 146 signals per run of runner-session.test.ts. On the CI runner the sibling vitest forks live in that pid band, so the Coverage job periodically lost one fork mid-file with no test attributed (issue #1824, 6 of the last 40 red CI runs). The group-signal write moves behind signalProcessGroupBestEffort in host-process.ts, next to signalPidsBestEffort, so the runner tests mock the signal seam in the same place they already mock the liveness reads. Refs #1824 * Revert "ci: w3-1824 experiment — trace fork signals and plant pid sentinels in the Coverage job" This reverts commit a8a0200. * test: refuse spawned kill/pkill writes too, and share the guard with the mutation lane Review of #1854 found three gaps in the first pass: - The guard intercepted process.kill only, so the spawned half of the same function family was unguarded: runner-disposal spawns `pkill -P <pid>` and `pkill -f 'xcodebuild.*AgentDeviceRunner.env.session-...'`, and request-router-open.test.ts fired that pattern kill twice per suite run. On a developer machine with a live Apple runner, `pnpm test` could reach it. The setup file now refuses kill/pkill/killall spawns with ENOENT — which the best-effort callers already tolerate — and records them the same way; that test stubs the Apple tool seam. - vitest.mutation.config.ts hard-coded its own setupFiles list, so the Stryker lane ran without the guard. SETUP_FILES is now exported from vitest.config.ts and imported there, next to the SUBPROCESS_STUB_TESTS import that already crossed the same boundary. - 'processes it spawned' meant direct children only; a grandchild started through a shell wrapper was refused with advice that did not fit. The docs and the failure message now say direct children and name the remedy. Synchronous spawns are no longer remembered as own pids: spawnSync and execFileSync have already exited when they return, so keeping their pids would license a signal to whatever inherits them next. Refs #1824 * test: end signal authority at child exit, and guard the promisified execFile path Re-review of #1854 found two holes in the guard itself, both the class it exists to close: - An async child's pid stayed authorized for the worker's lifetime after it was reaped. A pid is a claim on a process-table slot, and the kernel reissues that slot once it is free, so 'I spawned this pid once' licensed a signal to whatever holds it now. Authority now ends on exit/close. Cleanup that signals a child must gate on isProcessAlive, which is what the existing cleanup paths already do. - wrapSpawner copied execFile's original util.promisify.custom onto the wrapper, and promisify() resolves through that symbol instead of calling the function — so every promisified caller got an unguarded execFile, bypassing both the kill-binary refusal and child tracking. That path is now wrapped too. hermetic-signal-setup.test.ts covers both, plus the allowed cases they could regress into: a reaped child's pid is refused, a promisified execFile cannot smuggle a pkill, a promisified child is still tracked, a live foreign pid is refused, and signal 0 stays free. Reverting either fix reds three of them. Refs #1824
1 parent db5d26b commit e4c3b42

14 files changed

Lines changed: 498 additions & 110 deletions

‎docs/agents/testing.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,23 @@ the runtime seam existed; retiring `dispatchCommand('snapshot')` surfaced them o
100100
time. Do not add to that set, and when a command migrates (`docs/agents/adr-0019-unit.md`), its
101101
tests move to the runtime seam in the same PR.
102102

103+
Signals are hermetic too. A vitest worker may signal only itself and its own **direct children**;
104+
`src/__tests__/hermetic-signal-setup.ts` refuses every other process-table write and fails the
105+
sending test by name. It covers both ways out of the process, because the runner-disposal family
106+
uses both: `process.kill` (signal 0, the liveness probe, stays free) and a spawned
107+
`kill`/`pkill`/`killall`, whose `-P` and `-f` forms reach processes the worker never spawned at all.
108+
The refused pid is usually one the test made up (`child: { pid: 4242 }`), and on a real host that
109+
number can belong to anyone — on CI it periodically belonged to a sibling fork, which died mid-file
110+
with no test attributed ("Worker exited unexpectedly", #1824); a `pkill -f 'xcodebuild.*'` from a
111+
unit test would find a developer's live runner.
112+
113+
If a test drives a real kill path against a fabricated pid, mock the seam where it already mocks the
114+
liveness reads: `signalPidsBestEffort` / `signalProcessGroupBestEffort` in `src/utils/host-process.ts`
115+
for direct writes, the exec or tool-provider seam for a spawned `pkill`. Killing a daemon or Metro
116+
fixture the test itself spawned is fine — that pid is the worker's own. *Direct* is literal: a
117+
grandchild started through a shell or `npx` wrapper is not tracked, so signal the direct child, or
118+
its process group through the negative pid, rather than the grandchild's pid.
119+
103120
Keep tests behavioral. Do not assert shapes or cases TypeScript already proves.
104121

105122
A test added as a regression pin must be shown to fail without the change it pins — vacuity is the
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
import assert from 'node:assert/strict';
2+
import { execFile, spawn } from 'node:child_process';
3+
import { once } from 'node:events';
4+
import { promisify } from 'node:util';
5+
import { afterEach, test } from 'vitest';
6+
import { drainRefusedWritesForTest } from './hermetic-signal-setup.ts';
7+
8+
// The guard in hermetic-signal-setup.ts is what keeps a unit test from reaching
9+
// a process it does not own (#1824). These drive its refusal paths deliberately,
10+
// so each one drains the record the guard would otherwise fail the test with.
11+
12+
const execFileAsync = promisify(execFile);
13+
14+
afterEach(() => {
15+
assert.deepEqual(drainRefusedWritesForTest(), [], 'a test left an unasserted refusal behind');
16+
});
17+
18+
function spawnSleeper() {
19+
return spawn(process.execPath, ['-e', 'setTimeout(() => {}, 10_000)'], { stdio: 'ignore' });
20+
}
21+
22+
test('a live direct child may be signalled, and its process group with it', async () => {
23+
const child = spawnSleeper();
24+
const pid = child.pid;
25+
assert.ok(pid);
26+
27+
process.kill(pid, 'SIGTERM');
28+
await once(child, 'exit');
29+
30+
assert.deepEqual(drainRefusedWritesForTest(), []);
31+
});
32+
33+
test('authority ends when a child is reaped, so its recycled pid is refused', async () => {
34+
const child = spawnSleeper();
35+
const pid = child.pid;
36+
assert.ok(pid);
37+
child.kill('SIGKILL');
38+
await once(child, 'exit');
39+
40+
// The pid is now free for the kernel to hand to an unrelated process, so the
41+
// worker no longer owns it — even though this worker did spawn it once.
42+
assert.throws(
43+
() => process.kill(pid, 'SIGTERM'),
44+
(error: NodeJS.ErrnoException) =>
45+
error.code === 'ESRCH' && /did not spawn it/.test(error.message),
46+
);
47+
48+
const [record] = drainRefusedWritesForTest();
49+
assert.match(record ?? '', new RegExp(`SIGTERM -> pid ${pid}`));
50+
});
51+
52+
test('a live process this worker never spawned is refused', () => {
53+
// The vitest main process: alive, not ours.
54+
assert.throws(
55+
() => process.kill(process.ppid, 'SIGTERM'),
56+
(error: NodeJS.ErrnoException) => error.code === 'ESRCH',
57+
);
58+
assert.equal(drainRefusedWritesForTest().length, 1);
59+
});
60+
61+
test('a liveness probe stays free', () => {
62+
assert.equal(process.kill(process.ppid, 0), true);
63+
assert.deepEqual(drainRefusedWritesForTest(), []);
64+
});
65+
66+
test('a promisified execFile cannot smuggle a pkill past the guard', async () => {
67+
// promisify() resolves through the custom-promisify symbol rather than calling
68+
// the wrapper, so this path needs its own interception.
69+
await assert.rejects(
70+
execFileAsync('pkill', ['-f', 'xcodebuild.*AgentDeviceRunner']),
71+
(error: NodeJS.ErrnoException) =>
72+
error.code === 'ENOENT' && /Refusing to spawn/.test(error.message),
73+
);
74+
75+
const [record] = drainRefusedWritesForTest();
76+
assert.match(record ?? '', /spawn pkill -f xcodebuild/);
77+
});
78+
79+
test('a promisified execFile still tracks the child it starts', async () => {
80+
const pending = execFileAsync(process.execPath, ['-e', 'setTimeout(() => {}, 10_000)']);
81+
const pid = pending.child.pid;
82+
assert.ok(pid);
83+
84+
process.kill(pid, 'SIGKILL');
85+
await pending.catch(() => {});
86+
87+
assert.deepEqual(drainRefusedWritesForTest(), []);
88+
});
89+
90+
test('a spawned kill binary is refused whichever entry point starts it', () => {
91+
assert.throws(() => spawn('pkill', ['-TERM', '-P', '4242']), /Refusing to spawn/);
92+
assert.throws(() => execFile('/usr/bin/killall', ['node']), /Refusing to spawn/);
93+
assert.equal(drainRefusedWritesForTest().length, 2);
94+
});
Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
import childProcess from 'node:child_process';
2+
import module from 'node:module';
3+
import util from 'node:util';
4+
import { afterAll, afterEach, expect } from 'vitest';
5+
6+
// Unit tests must be hermetic with respect to the host's process table: a
7+
// worker may signal only itself and its own direct children. Anything else is
8+
// a pid the test made up (`child: { pid: 4242 }`), and on a real host that
9+
// number can belong to anyone — on CI it periodically belonged to a sibling
10+
// vitest fork, which died mid-file with no test attributed
11+
// ("Worker exited unexpectedly", #1824). Refusing the write here, in every
12+
// worker, turns that silent fork death into a named failure of the test that
13+
// sent it, on any host, deterministically.
14+
//
15+
// Both ways out of the process are covered, because the runner-disposal family
16+
// uses both: `process.kill` (including the negative pid that addresses a
17+
// child's process group) and a spawned `kill`/`pkill`/`killall`, whose `-P`
18+
// and `-f` forms reach processes this worker never spawned at all.
19+
//
20+
// Signal 0 is a liveness probe (`isProcessAlive`), not a signal; it stays free.
21+
// Tests that drive a real kill path against a fabricated pid mock the seam —
22+
// `signalPidsBestEffort` / `signalProcessGroupBestEffort` in
23+
// `src/utils/host-process.ts` for the direct writes, the exec/tool-provider
24+
// seam for a spawned `pkill` — the same way they already mock the liveness
25+
// reads.
26+
27+
/**
28+
* Direct, *live* children only.
29+
*
30+
* A grandchild (a server started through a shell or `npx` wrapper) is not
31+
* tracked, so signalling its pid is refused even though the test legitimately
32+
* owns it; signal the direct child, or its process group via the negative pid,
33+
* or mock the seam.
34+
*
35+
* Authority ends when the child is reaped: a pid is only ever a claim on a slot
36+
* in the host's process table, and the kernel hands that slot to someone else
37+
* once it is free. So synchronous spawns (`spawnSync`, `execFileSync`) are never
38+
* remembered — they have already exited when the call returns — and async
39+
* children are dropped on `exit`. Cleanup that signals a child must gate on
40+
* `isProcessAlive` (as `stopProcess` in `client-metro.test.ts` already does);
41+
* "I spawned this pid once" is not ownership of whatever holds it now, which is
42+
* the exact hazard this file exists to close.
43+
*/
44+
const ownPids = new Set<number>([process.pid]);
45+
46+
const KILL_BINARIES = new Set(['kill', 'pkill', 'killall']);
47+
48+
const refused: string[] = [];
49+
50+
function record(what: string): void {
51+
const state = expect.getState();
52+
const where = new Error().stack?.split('\n').slice(3, 7).join('\n') ?? '';
53+
refused.push(`${what} (${state.currentTestName ?? 'outside a test'})\n${where}`);
54+
}
55+
56+
function rememberChild(child: unknown): void {
57+
const handle = child as { pid?: unknown; once?: (event: string, cb: () => void) => void } | null;
58+
const pid = handle?.pid;
59+
if (typeof pid !== 'number') return;
60+
ownPids.add(pid);
61+
// Both events are attached because a child that is never spawned at all
62+
// ('error') emits close without exit.
63+
handle?.once?.('exit', () => ownPids.delete(pid));
64+
handle?.once?.('close', () => ownPids.delete(pid));
65+
}
66+
67+
function killBinaryName(command: unknown): string | undefined {
68+
// `exec` takes a shell command line; the rest take an executable path.
69+
const first = String(command).trim().split(/\s+/)[0] ?? '';
70+
const name = first.split('/').pop() ?? '';
71+
return KILL_BINARIES.has(name) ? name : undefined;
72+
}
73+
74+
function refuseSpawnedKill(command: unknown, args: unknown): Error | undefined {
75+
const name = killBinaryName(command);
76+
if (!name) return undefined;
77+
const rendered = [String(command), ...(Array.isArray(args) ? args.map(String) : [])].join(' ');
78+
record(`spawn ${rendered}`);
79+
// Shaped like the binary being unavailable, which every caller of these
80+
// best-effort kills already tolerates.
81+
const error = new Error(
82+
`Refusing to spawn \`${rendered}\`: a unit test may not signal processes through ${name} (see afterEach failure).`,
83+
) as NodeJS.ErrnoException;
84+
error.code = 'ENOENT';
85+
return error;
86+
}
87+
88+
function wrapSpawner<T extends (...args: never[]) => unknown>(real: T, remember: boolean): T {
89+
const wrapped = ((...args: Parameters<T>) => {
90+
const refusal = refuseSpawnedKill(args[0], args[1]);
91+
if (refusal) throw refusal;
92+
const child = real(...args);
93+
if (remember) rememberChild(child);
94+
return child;
95+
}) as unknown as T;
96+
// `util.promisify(execFile)` does not call the function at all: it resolves
97+
// through the custom-promisify symbol, so copying the original across would
98+
// hand every promisified caller an unguarded execFile. Wrap that path too.
99+
const custom = (real as unknown as Record<symbol, unknown>)[util.promisify.custom] as
100+
| ((...args: unknown[]) => unknown)
101+
| undefined;
102+
if (custom) {
103+
const wrappedCustom = (...args: unknown[]) => {
104+
const refusal = refuseSpawnedKill(args[0], args[1]);
105+
if (refusal) return Promise.reject(refusal);
106+
const promise = custom(...args);
107+
if (remember) rememberChild((promise as { child?: unknown } | null)?.child);
108+
return promise;
109+
};
110+
Object.defineProperty(wrapped, util.promisify.custom, { value: wrappedCustom });
111+
}
112+
return wrapped;
113+
}
114+
115+
for (const name of ['spawn', 'fork', 'exec', 'execFile'] as const) {
116+
(childProcess as unknown as Record<string, unknown>)[name] = wrapSpawner(
117+
childProcess[name],
118+
true,
119+
);
120+
}
121+
for (const name of ['spawnSync', 'execFileSync', 'execSync'] as const) {
122+
(childProcess as unknown as Record<string, unknown>)[name] = wrapSpawner(
123+
childProcess[name],
124+
false,
125+
);
126+
}
127+
// Named ESM imports of the builtin (`import { spawn } from 'node:child_process'`)
128+
// bind to the CJS exports only after they are re-synced.
129+
module.syncBuiltinESMExports();
130+
131+
const realKill = process.kill.bind(process);
132+
process.kill = ((pid: number, signal: string | number = 'SIGTERM') => {
133+
if (signal === 0 || ownPids.has(Math.abs(pid))) return realKill(pid, signal as NodeJS.Signals);
134+
record(`${String(signal)} -> pid ${pid}`);
135+
// Behave like a dead pid: the caller's ESRCH handling runs, no signal leaves
136+
// this worker, and the record above fails the test in afterEach below.
137+
const error = new Error(
138+
`Refusing to send ${String(signal)} to pid ${pid}: this vitest worker did not spawn it (see afterEach failure).`,
139+
) as NodeJS.ErrnoException;
140+
error.code = 'ESRCH';
141+
throw error;
142+
}) as typeof process.kill;
143+
144+
// Best-effort kill sites swallow the errors above, so the refusal is reported
145+
// here, attributed to the test (or file) that caused it, and never lost.
146+
function failOnRefusedSignals(scope: string): void {
147+
if (refused.length === 0) return;
148+
const count = refused.length;
149+
const report = refused.splice(0).join('\n');
150+
throw new Error(
151+
`${scope} tried to signal ${count} process(es) this vitest worker did not spawn. ` +
152+
'A unit test may signal only itself and its own direct children (and their process ' +
153+
'groups); mock the seam instead — signalPidsBestEffort / signalProcessGroupBestEffort in ' +
154+
'src/utils/host-process.ts, the exec or tool-provider seam for a spawned pkill, or ' +
155+
`vi.spyOn(process, 'kill'). If the pid is a descendant this test really owns, signal the ` +
156+
`direct child (or its group via the negative pid) rather than the grandchild.\n${report}`,
157+
);
158+
}
159+
160+
afterEach(() => failOnRefusedSignals('This test'));
161+
afterAll(() => failOnRefusedSignals('This file'));
162+
163+
/**
164+
* Takes the pending refusals so a test can assert on them without the hooks
165+
* above failing it. Only `hermetic-signal-setup.test.ts`, which drives the
166+
* guard's own refusal paths on purpose, should call this.
167+
*/
168+
export function drainRefusedWritesForTest(): string[] {
169+
return refused.splice(0);
170+
}

‎src/__tests__/test-file-size-ratchet.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,8 @@ const PINNED_TEST_FILE_LINES: Readonly<Record<string, number>> = Object.freeze({
3636
'src/__tests__/remote-connection.test.ts': 2973,
3737
'src/daemon/handlers/__tests__/snapshot-handler.test.ts': 2654,
3838
'src/commands/interaction/runtime/settle.test.ts': 2361,
39-
'src/platforms/apple/core/__tests__/runner-session.test.ts': 2083,
4039
'src/daemon/handlers/__tests__/session-replay-runtime-maestro.test.ts': 2031,
40+
'src/platforms/apple/core/__tests__/runner-session.test.ts': 2001,
4141
'src/utils/__tests__/daemon-client.test.ts': 1910,
4242
'src/utils/__tests__/output.test.ts': 1861,
4343
'src/platforms/android/__tests__/snapshot.test.ts': 1660,

‎src/daemon/__tests__/request-router-open.test.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,19 @@ vi.mock('../../utils/host-process.ts', async (importOriginal) => {
1212
const actual = await importOriginal<typeof import('../../utils/host-process.ts')>();
1313
return { ...actual, readProcessStartTime: vi.fn(() => 'test-process-start') };
1414
});
15+
// Opening a session runs the owned-lease cleanup, which pattern-kills stale
16+
// xcodebuild runners with a real `pkill -f`. The session id here is fabricated,
17+
// so on a host with a live Apple runner that write would reach a process this
18+
// test does not own; stub the tool seam the way the runner tests stub the
19+
// signal seam (#1824).
20+
vi.mock('../../platforms/apple/core/tool-provider.ts', async (importOriginal) => {
21+
const actual =
22+
await importOriginal<typeof import('../../platforms/apple/core/tool-provider.ts')>();
23+
return {
24+
...actual,
25+
runAppleToolCommand: vi.fn(async () => ({ exitCode: 0, stdout: '', stderr: '' })),
26+
};
27+
});
1528

1629
import { dispatchCommand } from '../../core/dispatch.ts';
1730
import {

‎src/platforms/apple/core/__tests__/runner-disposal.test.ts‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,13 +14,15 @@ const {
1414
mockRunAppleToolCommand,
1515
mockRunXcrun,
1616
mockSignalPidsBestEffort,
17+
mockSignalProcessGroupBestEffort,
1718
} = vi.hoisted(() => ({
1819
mockCleanupTempFile: vi.fn(),
1920
mockIsProcessAlive: vi.fn(),
2021
mockIsProcessGroupAlive: vi.fn(),
2122
mockRunAppleToolCommand: vi.fn(),
2223
mockRunXcrun: vi.fn(),
2324
mockSignalPidsBestEffort: vi.fn(),
25+
mockSignalProcessGroupBestEffort: vi.fn(),
2426
}));
2527

2628
vi.mock('../../../../utils/host-process.ts', async (importOriginal) => {
@@ -30,6 +32,7 @@ vi.mock('../../../../utils/host-process.ts', async (importOriginal) => {
3032
isProcessAlive: mockIsProcessAlive,
3133
isProcessGroupAlive: mockIsProcessGroupAlive,
3234
signalPidsBestEffort: mockSignalPidsBestEffort,
35+
signalProcessGroupBestEffort: mockSignalProcessGroupBestEffort,
3336
};
3437
});
3538

@@ -51,7 +54,6 @@ import { abortRunnerSessionsAndPrepProcesses } from '../runner/runner-disposal.t
5154

5255
beforeEach(() => {
5356
vi.useFakeTimers();
54-
vi.spyOn(process, 'kill').mockImplementation(() => true);
5557
mockIsProcessAlive.mockReturnValue(true);
5658
mockIsProcessGroupAlive.mockReturnValue(false);
5759
mockRunAppleToolCommand.mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' });
@@ -144,9 +146,8 @@ function makeRunnerSession(
144146
}
145147

146148
function runnerSignals(session: RunnerSession): NodeJS.Signals[] {
147-
return vi
148-
.mocked(process.kill)
149-
.mock.calls.filter(([pid]) => pid === -(session.child.pid ?? 0))
149+
return mockSignalProcessGroupBestEffort.mock.calls
150+
.filter(([pid]) => pid === session.child.pid)
150151
.map(([, signal]) => signal as NodeJS.Signals);
151152
}
152153

‎src/platforms/apple/core/__tests__/runner-request-cancellation.test.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ const {
1818
mockRunAppleToolCommand,
1919
mockRunCmdBackground,
2020
mockRunXcrun,
21+
mockSignalPidsBestEffort,
22+
mockSignalProcessGroupBestEffort,
2123
mockWaitForRunner,
2224
mockRedirectRelease,
2325
} = vi.hoisted(() => ({
@@ -32,6 +34,9 @@ const {
3234
mockRunAppleToolCommand: vi.fn(),
3335
mockRunCmdBackground: vi.fn(),
3436
mockRunXcrun: vi.fn(),
37+
// Runner child pids here are fabricated (4141..4444); see runner-session.test.ts.
38+
mockSignalPidsBestEffort: vi.fn(),
39+
mockSignalProcessGroupBestEffort: vi.fn(),
3540
mockWaitForRunner: vi.fn(),
3641
mockRedirectRelease: vi.fn(),
3742
}));
@@ -54,6 +59,8 @@ vi.mock('../../../../utils/host-process.ts', async () => {
5459
...actual,
5560
isProcessAlive: mockIsProcessAlive,
5661
isProcessGroupAlive: mockIsProcessGroupAlive,
62+
signalPidsBestEffort: mockSignalPidsBestEffort,
63+
signalProcessGroupBestEffort: mockSignalProcessGroupBestEffort,
5764
};
5865
});
5966

0 commit comments

Comments
 (0)