Skip to content

Commit c0c5ca7

Browse files
committed
fix(fuzz): run parser cases in a worker process, not the runner's thread (#2053)
The unit-lane corpus replay executed adversarial parser cases on worker threads of the Vitest worker running the test file. A fault in a worker thread ends its whole process, so a case that faulted killed the test runner: `[vitest-pool]: Worker forks emitted error / Worker exited unexpectedly`, with no test, file, or case named. Six of six Coverage deaths before #1994's split were this one file out of ~1100, and the uninstrumented second leg it created then lost the same file six more times in three days. Cases now run in a worker *process*. The two faults a case cannot report about itself are both classified from outside it: a case that never returns is a `hang` (unchanged), and one that ends the process it runs in is a new `crash` failure carrying the exit code or signal and the tail of the worker's stderr — the death certificate the lane used to lose. A sixth self-check target seeds that kind, so a regression in reporting it fails the harness self-check like every other kind.
1 parent 8af2660 commit c0c5ca7

9 files changed

Lines changed: 195 additions & 63 deletions

File tree

‎.github/workflows/replays-nightly.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,10 @@ jobs:
3838
# 38,000 cases/target holds the job's wall-clock at the pre-B2 five-target/50k budget now that
3939
# seven targets share it: measured on a quiet host, 7x38k = 16.5s against 5x50k = 16.6-16.9s.
4040
# Chosen by measurement rather than rounding — 33k would have been 2s cheaper but cost the
41-
# five untouched targets a third of their depth for no wall-clock reason.
41+
# five untouched targets a third of their depth for no wall-clock reason. The count is
42+
# unchanged by #2053; moving case execution out of process cost that run about 3s (17.8s ->
43+
# 20.9s, best of three on one host), which buys a case that faults its process a named
44+
# `crash` failure instead of a dead caller.
4245
nightly-parser-fuzz:
4346
name: Parser Fuzz Lane
4447
runs-on: ubuntu-latest

‎docs/agents/testing.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -115,8 +115,11 @@ another example. Keep examples for a real past bug or a named decision. Reuse `P
115115
budgets so property files stay inside the unit slow-test gate.
116116

117117
Parser fuzz targets live in `scripts/fuzz/targets.ts`. Validation generators carry the invalid
118-
outcome they planted, so silent acceptance and wrong error codes are failures. Promote a discovered
119-
case with the command the harness prints — never hand-copy an unshrunk input.
118+
outcome they planted, so silent acceptance and wrong error codes are failures. Cases run in a
119+
worker process, so the two faults a case cannot report about itself — never returning, and killing
120+
the process it runs in — are reported as `hang` and `crash` against the exact input rather than
121+
taking the caller down with them. Promote a discovered case with the command the harness prints —
122+
never hand-copy an unshrunk input.
120123

121124
Mutation is report-only and limited to the registry in `scripts/mutation/modules.ts`. It measures
122125
whether tests distinguish changed decision logic. Do not infer redundancy from line coverage alone.

‎scripts/fuzz/corpus-replay.test.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,10 @@
22
//
33
// The nightly lane finds cases; this replays every case it ever found so a regression fails in
44
// seconds on a PR instead of a night later. Cases go through the same worker-backed watchdog the
5-
// nightly lane uses: a promoted hang case must fail this test against its per-case budget, not
6-
// wedge the unit job until the CI timeout.
5+
// nightly lane uses, and for the same reason they run in a worker *process* (#2053): a promoted
6+
// hang case must fail this test against its per-case budget rather than wedge the unit job, and
7+
// a case that faults the process it runs in must fail this test rather than kill the Vitest
8+
// worker running it — which reports no test, no file, and no case.
79

810
import fc from 'fast-check';
911
import { describe, expect, it } from 'vitest';

‎scripts/fuzz/execute.ts‎

Lines changed: 111 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,84 +1,158 @@
1-
// Case execution with hang detection for the parser fuzz lane (#1414).
1+
// Case execution with hang and crash detection for the parser fuzz lane (#1414).
22
//
3-
// Cases run one at a time in a worker thread: a synchronous parser that never returns cannot be
4-
// timed out from inside its own tick, so the budget is enforced from *another* thread, which can
5-
// terminate the wedged one and attribute the stall to the exact input. Cases go over the wire
6-
// individually (rather than as one batch) because fast-check drives the loop — it decides the next
7-
// input, including the shrink candidates it derives from a failing one.
3+
// Cases run one at a time in a child process, and the budget is enforced from the parent. Two
4+
// faults a case can produce cannot be observed from inside the context running it: a synchronous
5+
// parser that never returns cannot be timed out from its own tick, and a parser that faults the
6+
// process — a native crash, an out-of-memory abort — leaves no stack to catch. Both are
7+
// attributed here, to the exact input, by watching a process the parent can outlive.
8+
//
9+
// It is a process rather than a worker thread because a thread cannot contain the second fault:
10+
// a fault in a worker thread takes its whole process down, and that process is the caller's —
11+
// for the unit-lane replay, the Vitest worker running the test file, which then dies with no
12+
// test, file, or case named (#2053). Cases go over the wire individually (rather than as one
13+
// batch) because fast-check drives the loop — it decides the next input, including the shrink
14+
// candidates it derives from a failing one.
815

16+
import { fork, type ChildProcess } from 'node:child_process';
917
import path from 'node:path';
1018
import { fileURLToPath } from 'node:url';
11-
import { Worker } from 'node:worker_threads';
1219
import type { FuzzFailure } from './invariant.ts';
1320
import type { FuzzTarget } from './target-types.ts';
14-
import type { FuzzWorkerData, FuzzWorkerMessage } from './worker.ts';
21+
import type { FuzzWorkerMessage } from './worker.ts';
1522

1623
const WORKER_PATH = path.join(path.dirname(fileURLToPath(import.meta.url)), 'worker.ts');
1724

1825
/** Bound on worker startup only; a per-case budget must never be charged for it. */
1926
const STARTUP_BUDGET_MS = 60_000;
2027

28+
/** How much of a dead worker's output a crash failure quotes. Enough for a V8 fatal banner. */
29+
const DEATH_OUTPUT_LIMIT = 2_000;
30+
2131
export type CaseRunner = {
2232
/** The failure this input violates the invariant with, or `null` when it holds. */
2333
run: (input: string) => Promise<FuzzFailure | null>;
2434
close: () => Promise<void>;
2535
};
2636

27-
type Session = { worker: Worker; ready: Promise<void> };
37+
type Session = {
38+
child: ChildProcess;
39+
/** Settles when the worker has finished importing the parsers. */
40+
ready: Promise<void>;
41+
/**
42+
* How the process died, or `null` while it is alive — the death certificate a crash failure
43+
* quotes. A reader rather than a field because it is written by the session's own listeners.
44+
*/
45+
death: () => string | null;
46+
};
47+
48+
function describeDeath(code: number | null, signal: NodeJS.Signals | null, output: string): string {
49+
const how = signal === null ? `exit code ${code}` : `signal ${signal}`;
50+
const tail = output.trim();
51+
return tail === '' ? how : `${how}: ${tail}`;
52+
}
2853

29-
/** A worker plus the promise that settles when it has finished importing the parsers. */
54+
/** A worker process plus the promise that settles when it has finished importing the parsers. */
3055
function startSession(targetName: string): Session {
31-
const workerData: FuzzWorkerData = { targetName };
32-
// Type stripping is requested explicitly rather than inherited: under Vitest the parent's
33-
// execArgv carries no such flag, and the worker is a plain `.ts` file Node must strip itself.
34-
// The warning is silenced because a restarted worker re-emits it — one per hang buries the report.
35-
const worker = new Worker(WORKER_PATH, {
36-
workerData,
56+
const child = fork(WORKER_PATH, [targetName], {
57+
// Type stripping is requested explicitly rather than inherited: under Vitest the parent's
58+
// execArgv carries no such flag, and the worker is a plain `.ts` file Node must strip itself.
59+
// The warning is silenced because a restarted worker re-emits it — one per hang buries the report.
3760
execArgv: ['--experimental-strip-types', '--disable-warning=ExperimentalWarning'],
61+
// stderr is piped rather than inherited so a fatal fault — which writes to the descriptor and
62+
// so escapes the worker's own silencing — is quoted in the failure instead of the lane's log.
63+
stdio: ['ignore', 'inherit', 'pipe', 'ipc'],
64+
});
65+
let death: string | null = null;
66+
let output = '';
67+
child.stderr?.setEncoding('utf8');
68+
child.stderr?.on('data', (chunk: string) => {
69+
output = (output + chunk).slice(-DEATH_OUTPUT_LIMIT);
70+
});
71+
// `close` rather than `exit`: it fires once the piped stderr has drained too, so the death is
72+
// recorded with the output that explains it.
73+
child.once('close', (code: number | null, signal: NodeJS.Signals | null) => {
74+
death = describeDeath(code, signal, output);
3875
});
3976
const ready = new Promise<void>((resolve, reject) => {
4077
const timer = setTimeout(
4178
() => reject(new Error(`fuzz worker did not start within ${STARTUP_BUDGET_MS}ms`)),
4279
STARTUP_BUDGET_MS,
4380
);
44-
worker.once('message', (message: FuzzWorkerMessage) => {
81+
child.once('message', (message: FuzzWorkerMessage) => {
4582
clearTimeout(timer);
4683
if (message.kind === 'ready') resolve();
4784
else reject(new Error(`unexpected first worker message: ${message.kind}`));
4885
});
49-
worker.once('error', (error) => {
86+
child.once('error', (error) => {
5087
clearTimeout(timer);
5188
reject(error);
5289
});
90+
child.once('close', () => {
91+
clearTimeout(timer);
92+
reject(new Error(`fuzz worker died before it was ready (${death})`));
93+
});
5394
});
54-
return { worker, ready };
95+
return { child, ready, death: () => death };
96+
}
97+
98+
/** Ends a worker process, and resolves once it is gone. Already-dead sessions are a no-op. */
99+
async function endSession(session: Session): Promise<void> {
100+
if (session.death() !== null) return;
101+
const gone = new Promise<void>((resolve) => session.child.once('close', () => resolve()));
102+
// SIGKILL, not SIGTERM: the session is ended when a case wedged the worker, and a process
103+
// spinning inside a synchronous parser is exactly the one a catchable signal cannot reach.
104+
session.child.kill('SIGKILL');
105+
await gone;
55106
}
56107

57108
/**
58109
* Runs one case on a started worker. Startup is already awaited by the caller, so the budget below
59110
* covers parser time only — the reason a slow import can never be misreported as a parser hang.
111+
*
112+
* `restart` reports that the worker is no longer usable: it is wedged in the case that just hung,
113+
* or it is dead.
60114
*/
61115
function runOnSession(
62116
session: Session,
63117
target: FuzzTarget,
64118
input: string,
65119
caseTimeoutMs: number,
66-
): Promise<{ failure: FuzzFailure | null; hung: boolean }> {
120+
): Promise<{ failure: FuzzFailure | null; restart: boolean }> {
121+
const crashed = (): { failure: FuzzFailure; restart: boolean } => ({
122+
failure: {
123+
target: target.name,
124+
input,
125+
kind: 'crash',
126+
detail: `case killed the worker process (${session.death()})`,
127+
},
128+
restart: true,
129+
});
130+
// A worker that died between cases is reported against the case that meets it, rather than
131+
// through the ERR_IPC_CHANNEL_CLOSED that sending to it would raise: the harness classifies
132+
// every death of a worker it was using, and an opaque transport error is not a classification.
133+
if (session.death() !== null) return Promise.resolve(crashed());
67134
return new Promise((resolve, reject) => {
68-
const settle = (failure: FuzzFailure | null, hung: boolean) => {
135+
const detach = () => {
69136
clearTimeout(timer);
70-
session.worker.off('message', onMessage);
71-
session.worker.off('error', onError);
72-
resolve({ failure, hung });
137+
session.child.off('message', onMessage);
138+
session.child.off('error', onError);
139+
session.child.off('close', onClose);
140+
};
141+
const settle = (failure: FuzzFailure | null, restart: boolean) => {
142+
detach();
143+
resolve({ failure, restart });
73144
};
74145
const onMessage = (message: FuzzWorkerMessage) => {
75146
if (message.kind === 'result') settle(message.failure, false);
76147
};
77148
const onError = (error: Error) => {
78-
clearTimeout(timer);
79-
session.worker.off('message', onMessage);
149+
detach();
80150
reject(error);
81151
};
152+
const onClose = () => {
153+
const { failure, restart } = crashed();
154+
settle(failure, restart);
155+
};
82156
const timer = setTimeout(() => {
83157
settle(
84158
{
@@ -90,16 +164,17 @@ function runOnSession(
90164
true,
91165
);
92166
}, caseTimeoutMs);
93-
session.worker.on('message', onMessage);
94-
session.worker.once('error', onError);
95-
session.worker.postMessage({ kind: 'case', input });
167+
session.child.on('message', onMessage);
168+
session.child.once('error', onError);
169+
session.child.once('close', onClose);
170+
session.child.send({ kind: 'case', input });
96171
});
97172
}
98173

99174
/**
100-
* A worker-backed runner for one target. A hang leaves the thread wedged in its own loop, so the
101-
* runner terminates it and starts a fresh one for the next case — otherwise a single hang would
102-
* silently turn every later case into a hang too.
175+
* A worker-backed runner for one target. A hang leaves the worker wedged in its own loop and a
176+
* crash leaves no worker at all, so the runner ends the session and starts a fresh one for the
177+
* next case — otherwise a single such case would silently turn every later case into one too.
103178
*/
104179
export async function createCaseRunner(
105180
target: FuzzTarget,
@@ -113,17 +188,17 @@ export async function createCaseRunner(
113188
run: async (input) => {
114189
if (closed) throw new Error('fuzz case runner is closed');
115190
await session.ready;
116-
const { failure, hung } = await runOnSession(session, target, input, caseTimeoutMs);
117-
if (hung) {
118-
await session.worker.terminate();
191+
const { failure, restart } = await runOnSession(session, target, input, caseTimeoutMs);
192+
if (restart) {
193+
await endSession(session);
119194
session = startSession(target.name);
120195
await session.ready;
121196
}
122197
return failure;
123198
},
124199
close: async () => {
125200
closed = true;
126-
await session.worker.terminate();
201+
await endSession(session);
127202
},
128203
};
129204
}

‎scripts/fuzz/harness.test.ts‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import { fileURLToPath } from 'node:url';
1414
import { describe, expect, it } from 'vitest';
1515
import { LANE_ENVELOPE_SCHEMA_VERSION } from '../lib/lane-envelope.ts';
1616
import { CASE_GENERATION_INPUTS } from './envelope.ts';
17+
import { runCases } from './execute.ts';
1718
import { checkCase } from './invariant.ts';
1819
import { SELF_CHECK_TARGETS } from './self-check-targets.ts';
1920

@@ -69,7 +70,7 @@ describe('fuzz invariant classifier', () => {
6970
});
7071

7172
describe('fuzz harness self-check', () => {
72-
// One run asserts both the report and its envelope: a second full self-check would cost five
73+
// One run asserts both the report and its envelope: a second full self-check would cost six
7374
// more real worker startups (#1823) for no new signal.
7475
it('catches every seeded violation kind and writes the self-check envelope', () => {
7576
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fuzz-selfcheck-'));
@@ -83,17 +84,36 @@ describe('fuzz harness self-check', () => {
8384
expect(stdout).toContain('ok self-check-untyped-throw: expected untyped-throw');
8485
expect(stdout).toContain('ok self-check-empty-hint: expected empty-hint');
8586
expect(stdout).toContain('ok self-check-hang: expected hang, got hang');
87+
expect(stdout).toContain('ok self-check-crash: expected crash, got crash');
8688
expect(stdout).toContain('ok self-check-silent-accept: expected silent-accept');
8789
expect(stdout).toContain('ok self-check-wrong-code: expected wrong-code');
8890
expect(status).toBe(0);
8991
const envelope = JSON.parse(fs.readFileSync(path.join(dir, 'run-envelope.json'), 'utf8'));
9092
expect(envelope.result).toBe('pass');
9193
expect(envelope.data.mode).toBe('self-check');
92-
expect(envelope.data.targetRuns).toHaveLength(5);
94+
expect(envelope.data.targetRuns).toHaveLength(6);
9395
fs.rmSync(dir, { recursive: true, force: true });
9496
}, 30_000);
9597
});
9698

99+
describe('case crash containment', () => {
100+
// #2053: a case that faults the process it runs in must take only the worker with it. This
101+
// test runs the runner in-process on purpose — it *is* the caller the lane must protect, so a
102+
// runner that executed cases on this thread would kill this Vitest worker instead of failing,
103+
// and the file would vanish from the run with nothing attributed.
104+
it('attributes a worker death to the case and survives it', async () => {
105+
const failures = await runCases(targetNamed('self-check-crash'), ['case'], 5_000);
106+
expect(failures.map((failure) => failure.kind)).toEqual(['crash']);
107+
// The death certificate the lane used to lose: how the worker ended, quoted with the case.
108+
expect(failures[0]?.detail).toContain('exit code 97');
109+
}, 30_000);
110+
111+
it('keeps running cases after one kills the worker', async () => {
112+
const failures = await runCases(targetNamed('self-check-crash'), ['first', 'second'], 5_000);
113+
expect(failures.map((failure) => failure.input)).toEqual(['first', 'second']);
114+
}, 30_000);
115+
});
116+
97117
describe('worker startup budget', () => {
98118
// The watchdog used to start before the worker reported ready, so thread spawn plus type
99119
// stripping plus parser imports — hundreds of milliseconds — was charged to the first case and

‎scripts/fuzz/self-check-targets.ts‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,22 @@ export const SELF_CHECK_TARGETS: readonly FuzzTarget[] = [
2929
},
3030
seeds: SEEDS,
3131
},
32+
{
33+
// The kind no in-process classifier can produce: the case ends the process it runs in, so
34+
// only a runner watching from another process reports anything at all. `process.exit` stands
35+
// in for the real thing (a native fault) because it is deterministic and portable — the
36+
// runner classifies any death of a worker with a case in flight, whatever ended it.
37+
name: 'self-check-crash',
38+
description: 'ends the process mid-case (expects a crash failure)',
39+
run: () => process.exit(97),
40+
seeds: SEEDS,
41+
},
3242
{
3343
name: 'self-check-hang',
3444
description: 'never returns (expects a hang failure)',
3545
run: () => {
3646
for (;;) {
37-
// Spin forever: only the out-of-thread watchdog can end this case.
47+
// Spin forever: only the out-of-process watchdog can end this case.
3848
}
3949
},
4050
seeds: SEEDS,
@@ -78,6 +88,7 @@ export const SELF_CHECK_EXPECTATIONS = {
7888
'self-check-untyped-throw': 'untyped-throw',
7989
'self-check-empty-hint': 'empty-hint',
8090
'self-check-hang': 'hang',
91+
'self-check-crash': 'crash',
8192
'self-check-silent-accept': 'silent-accept',
8293
'self-check-wrong-code': 'wrong-code',
8394
} as const satisfies Record<SelfCheckTargetName, string>;

‎scripts/fuzz/target-types.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,24 @@ export type SelfCheckTargetName =
2020
| 'self-check-empty-hint'
2121
| 'self-check-hang'
2222
| 'self-check-silent-accept'
23-
| 'self-check-wrong-code';
23+
| 'self-check-wrong-code'
24+
| 'self-check-crash';
2425

2526
export type FuzzTargetName = ParserTargetName | SelfCheckTargetName;
2627

2728
/**
2829
* `silent-accept` and `wrong-code` only arise from validation targets, whose cases carry an
2930
* expected outcome (validation-case.ts); the classic targets can only produce the first three.
31+
*
32+
* `hang` and `crash` are the two the case itself cannot report — a parser that never returns
33+
* cannot time itself out, and one that faults the process leaves no stack to catch. Both are
34+
* classified by the runner, from outside the process the case runs in (execute.ts).
3035
*/
3136
export type FuzzFailureKind =
3237
| 'untyped-throw'
3338
| 'empty-hint'
3439
| 'hang'
40+
| 'crash'
3541
| 'silent-accept'
3642
| 'wrong-code';
3743

0 commit comments

Comments
 (0)